I have a Java rich client application that registers a durable subscription on a remote HornetQ JMS Topic on startup.
However if the server restarts the connection is lost and can only be restored by restarting the client application.
This leads to confusing situations where JMS messages are not received and after a client restart a lot of messages are received at once.
A simple solution to restore connection would be to run a timer to periodically check if the connection is still valid and try to reconnect otherwise.
Alternatively the server could send a heartbeat to the client and try to reconnect if no heartbeat is received after a certain period (like mentioned in this answer).
But both seem like clumsy approaches to this problem and therefore I would like to know if there is a better solution to find out that the connection is not available anymore?
To get notified on the disconnection you have to register an ExceptionListener on your TopicConnection before you start the connection.
private void subscribe() throws JMSException {
// get context / create connection and session / etc.
TopicConnection connection = ...
connection.setExceptionListener(this::handleExceptions);
connection.start();
}
In the ExceptionListener you can check the error code of the received JMSException. (The error codes are vendor-specific)
In the case of HornetQ the error code DISCONNECT is received after the connection is lost.
private static final String HORNETQ_DISCONNECT_ERROR_CODE = "DISCONNECT";
private void handleExceptions(final JMSException jmsException) {
final String errorCode = jmsException.getErrorCode();
if (HORNETQ_DISCONNECT_ERROR_CODE.equals(errorCode)) {
tryConnect();
}
}
Then you can start a self-canceling Timer that tries to reconnect every x seconds until it succeeds.
private static final long SUBSCRIBE_RETRY_TIME_OUT_IN_MILLIS = 60000;
private void tryConnect() {
final Timer timer = new Timer("JMS-Topic-Reconnection-Timer", true);
final TimerTask timerTask = new TimerTask() {
#Override
public void run() {
try {
subscribe();
// cancel the timer, after the subscription succeeds
timer.cancel();
}
catch (final Exception e) {
logger.info("reconnect to jms topic failed: {}", e.getMessage());
}
}
};
timer.schedule(timerTask, SUBSCRIBE_RETRY_TIME_OUT_IN_MILLIS, SUBSCRIBE_RETRY_TIME_OUT_IN_MILLIS);
}
Related
At the end of my Spark app (Containerized Spark v2.4.7), I publish a message to RabbitMQ. The app runs successfully and even the message is published to my containerized RabbitMQ. The problem is that the process doesn't really finish... I need to ctrl c from the terminal to abort the process.
Loglines that I write to console after publishing message are written, which means the process didn't stuck in the RabbitMQ client.
I tried to close the RabbitMQ channel and connection, but it didn't help.
Also, I tried to close the SparkSession at the end, but it didn't help either.
I wrote a test with Junit where I create the same queue with the same configuration and write the same message with the same client configuration and it finishes successfully, not stuck or anything.
My RabbitMQ publishing implementation:
private Connection getConnection() throws KeyManagementException, NoSuchAlgorithmException, URISyntaxException, IOException, TimeoutException{
ConnectionFactory factory = new ConnectionFactory();
String uri = getURI();
factory.setUri(new URI(uri));
factory.setConnectionTimeout(10000);
return factory.newConnection();
}
#Override
public void publish(String msg) {
try{
Connection connection = getConnection();
Channel channel = connection.createChannel();
channel.queueDeclare("rabbitQueue", true, false, false, null);
channel.basicPublish("exchange", "rabbitKey", null, msg.getBytes());
} catch (Exception ex){
logger.error("Error, failed to create connection to RabbitMQ", ex);
}
}
What should be the reason that my process doesn't finish?
Thanks!
According to the specs an HTTP server should send a 408 if it hasn't received a request in a certain time. This is a bit unintuitive as it means you can send a response without having received a request. One purpose is to kill long-lived keep-alive HTTP 1.1 connections that clients haven't closed.
To do this, I added an IdleStateEvent event and in there:
DefaultFullHttpResponse resp = new DefaultFullHttpResponse(HTTP_1_1,
HttpResponseStatus.REQUEST_TIMEOUT);
resp.headers().set(HttpHeaderNames.CONNECTION, HttpHeaderValues.CLOSE);
ctx.writeAndFlush(resp)
.addListener(future -> {
System.out.println("Finished " + future.cause());
})
.addListener(ChannelFutureListener.CLOSE);
And the output:
Finished io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: cannot send more responses than requests
Is there a way to do this in Netty? Or a recommended way to close idle HTTP 1.1 connections?
According to the Netty javadoc you can use the IdleStateHandler class to close idle connections. Apparently, this handler will trigger an IdleStateEvent when a connection has no reads, or no writes, or both, for a period of time. This event can then be used to trigger shutting down a connection ... or to do other things.
The following example is copied from the javadoc:
// An example that sends a ping message when there is no outbound traffic
// for 30 seconds. The connection is closed when there is no inbound traffic
// for 60 seconds.
public class MyChannelInitializer extends ChannelInitializer<Channel> {
#Override
public void initChannel(Channel channel) {
channel.pipeline().addLast("idleStateHandler", new IdleStateHandler(60, 30, 0));
channel.pipeline().addLast("myHandler", new MyHandler());
}
}
// Handler should handle the IdleStateEvent triggered by IdleStateHandler.
public class MyHandler extends ChannelDuplexHandler {
#Override
public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
if (evt instanceof IdleStateEvent) {
IdleStateEvent e = (IdleStateEvent) evt;
if (e.state() == IdleState.READER_IDLE) {
ctx.close();
} else if (e.state() == IdleState.WRITER_IDLE) {
ctx.writeAndFlush(new PingMessage());
}
}
}
}
ServerBootstrap bootstrap = ...;
...
bootstrap.childHandler(new MyChannelInitializer());
...
Note: according to this Q&A, the IdleStateHandler should be the first handler in the pipeline.
I have the following piece of code to start a basic Embedded Grizzly server running with Jersey.
private static void startServer() {
ServerResourceConfiguration configuration = new ServerResourceConfiguration();
HttpServer server = GrizzlyHttpServerFactory.createHttpServer(
URI.create(BASE_URI),
configuration,
false,
null,
false);
server.start();
if (System.in.read() > -2) {
server.shutdownNow();
}
}
This does not look like production level way to stop a server.
What is the best practice to gracefully shut it down ?
I guess a terminal command of some sort. Killing the process would work but it is not very graceful.
I am using Gradle on this project and runs the server with the gradle run command.
Could a Gradle task do the job?
Also I have seen this about gracefully terminating a grizzly transport:
http://grizzly-nio.net/2013/08/gracefully-terminating-a-grizzly-transport/
But I am not sure if I would need to use it. I don't understand how to use it.
EDIT: I came across this post:
https://stackoverflow.com/a/15391081/3982755
Is that an acceptable way to terminate an Http server in a production environment?
There is no answer so I will post my own, I implemented it with a Shutdown Hook and it works very well.
The server will wait for all connections to terminate before shutting down.
To avoid getting blocked for ever if a connection never terminates, we set a grace period(60 seconds)
After the grace period the server will force termination of all connections
Here is the code for the hook to be run when the server receives a SIGINT or SIGTERM signal.
public class GrizzlyServerShutdownHookThread extends Thread {
public static final String THREAD_NAME = "Grizzly Server Shutdown Hook";
public static final int GRACE_PERIOD = 60;
public static final TimeUnit GRACE_PERIOD_TIME_UNIT = TimeUnit.SECONDS;
private final HttpServer server;
/**
* #param server The server to shut down
*/
public GrizzlyServerShutdownHookThread(HttpServer server) {
this.server = server;
setName(THREAD_NAME);
}
#Override
public void run() {
LOG.info("Running Grizzly Server Shutdown Hook.");
LOG.info("Shutting down server.");
GrizzlyFuture<HttpServer> future = server.shutdown(GRACE_PERIOD, GRACE_PERIOD_TIME_UNIT);
try {
LOG.info(format("Waiting for server to shut down... Grace period is %s %s", GRACE_PERIOD, GRACE_PERIOD_TIME_UNIT));
future.get();
} catch(InterruptedException | ExecutionException e) {
LOG.error("Error while shutting down server.", e);
}
LOG.info("Server stopped.");
}
}
Then I register the Hook into the RunTime object this way when I setup the server:
Runtime.getRuntime().addShutdownHook(
new GrizzlyServerShutdownHookThread(server)
);
And finally, I start the server this way:
try {
server.start();
} catch (IOException e) {
throw new RuntimeException(e);
}
// wait for a SIGINT (Ctrl+c) signal to shut down
try {
LOG.info("Press CTRL^C to exit..");
Thread.currentThread().join();
} catch(InterruptedException e) {
throw new RuntimeException(e);
}
Background
I have implemented an adapter interface using the RPC protocol, but recently have been tasked with implementing the interface using a WebSocket listener. With RPC, I was easily able to start an RPC listener thread to listen for events on a separate thread, but I'm not finding it so simple when it comes to JSR356.
The Question
I'm attempting to implement a Java WebSocket ClientEndpoint that connects to a subscription URI, but I want to do so in a manner that utilizes multi-threading. I've been having a hard time finding any examples where multi-threading is needed from a client endpoint perspective. Is this even possible?
I need the WebSocket message handler to handle messages without blocking the main thread. I have not implemented the message handler yet because I'm not sure how to go about creating it in a way to accomplish what I want. Can anyone help point me in a better direction? Here's what I have so far:
EventHandler.java
#ClientEndpoint
public class EventHandler {
private URI subscriptionURI;
private Session clientSession;
public EventHandler(URI subscriptionURI) throws URISyntaxException {
this.subscriptionURI = subscriptionURI;
}
/**
* Attempts to connect to the CADI WebSocket server.
* #throws Exception
*/
public void connect() throws Exception {
// Grab the WebSocket container and attempt to connect to the subscription URI
WebSocketContainer container = ContainerProvider.getWebSocketContainer();
container.connectToServer(this, subscriptionURI);
}
/**
* Closes the CADI WebSocket client session.
*/
public void close() {
try {
// Close the client session if it is open
if(clientSession != null && clientSession.isOpen())
clientSession.close();
}
catch(Exception e) {
LogMaster.getErrorLogger().error("Could not close the WebSocket client session. It may have been closed already.", e);
}
}
#OnOpen
public void socketOpened(Session session) {
this.clientSession = session;
}
}
Here is how I start a new thread to connect to the WebSocket. What are the implications of this, though? Are subsequent messages received on the WebSocket going to block the main thread still?
EventHandler eventHandler = new EventHandler(new URI("wss://localhost/Example"));
new Thread()
{
#Override
public void run() {
try {
eventHandler.connect();
}
catch (Exception e) {
LogMaster.getErrorLogger().error("Could not start EventHandler.", e);
}
}
}.start();
I am using jersey to implement a SSE scenario.
The server keeps connections alive. And push data to clients periodically.
In my scenario, there is a connection limit, only a certain number of clients can subscribe to the server at the same time.
So when a new client is trying to subscribe, I do a check(EventOutput.isClosed) to see if any old connections are not active anymore, so they can make room for new connections.
But the result of EventOutput.isClosed is always false, unless the client explicitly calls close of EventSource. This means that if a client drops accidentally(power outage or internet cutoff), it's still hogging the connection, and new clients can not subscribe.
Is there a work around for this?
#CuiPengFei,
So in my travels trying to find an answer to this myself I stumbled upon a repository that explains how to handle gracefully cleaning up the connections from disconnected clients.
The encapsulate all of the SSE EventOutput logic into a Service/Manager. In this they spin up a thread that checks to see if the EventOutput has been closed by the client. If so they formally close the connection (EventOutput#close()). If not they try to write to the stream. If it throws an Exception then the client has disconnected without closing and it handles closing it. If the write is successful then the EventOutput is returned to the pool as it is still an active connection.
The repo (and the actual class) are available here. Ive also included the class without imports below in case the repo is ever removed.
Note that they bind this to a Singleton. The store should be globally unique.
public class SseWriteManager {
private final ConcurrentHashMap<String, EventOutput> connectionMap = new ConcurrentHashMap<>();
private final ScheduledExecutorService messageExecutorService;
private final Logger logger = LoggerFactory.getLogger(SseWriteManager.class);
public SseWriteManager() {
messageExecutorService = Executors.newScheduledThreadPool(1);
messageExecutorService.scheduleWithFixedDelay(new messageProcessor(), 0, 5, TimeUnit.SECONDS);
}
public void addSseConnection(String id, EventOutput eventOutput) {
logger.info("adding connection for id={}.", id);
connectionMap.put(id, eventOutput);
}
private class messageProcessor implements Runnable {
#Override
public void run() {
try {
Iterator<Map.Entry<String, EventOutput>> iterator = connectionMap.entrySet().iterator();
while (iterator.hasNext()) {
boolean remove = false;
Map.Entry<String, EventOutput> entry = iterator.next();
EventOutput eventOutput = entry.getValue();
if (eventOutput != null) {
if (eventOutput.isClosed()) {
remove = true;
} else {
try {
logger.info("writing to id={}.", entry.getKey());
eventOutput.write(new OutboundEvent.Builder().name("custom-message").data(String.class, "EOM").build());
} catch (Exception ex) {
logger.info(String.format("write failed to id=%s.", entry.getKey()), ex);
remove = true;
}
}
}
if (remove) {
// we are removing the eventOutput. close it is if it not already closed.
if (!eventOutput.isClosed()) {
try {
eventOutput.close();
} catch (Exception ex) {
// do nothing.
}
}
iterator.remove();
}
}
} catch (Exception ex) {
logger.error("messageProcessor.run threw exception.", ex);
}
}
}
public void shutdown() {
if (messageExecutorService != null && !messageExecutorService.isShutdown()) {
logger.info("SseWriteManager.shutdown: calling messageExecutorService.shutdown.");
messageExecutorService.shutdown();
} else {
logger.info("SseWriteManager.shutdown: messageExecutorService == null || messageExecutorService.isShutdown().");
}
}}
Wanted to provide an update on this:
What was happening is that the eventSource on the client side (js) never got into readyState '1' unless we did a broadcast as soon as a new subscription was added. Even in this state the client could receive data pushed from the server. Adding call to do a broadcast of a simple "OK" message helped kicking the eventSource into readyState 1.
On closing the connection from the client side; to be pro-active in cleaning up resources, just closing the eventSource on the client side doesn't help. We must make another ajax call to the server to force the server to do a broadcast. When the broadcast is forced, jersey will clean up the connections that are no longer alive and will in-turn release resources (Connections in CLOSE_WAIT). If not a connection will linger in CLOSE_WAIT till the next broadcast happens.