Simple scenario:
A lower level class A that extends SimpleChannelUpstreamHandler. This class is the workhorse to send the message and received the response.
A top level class B that can be used by other part of the system to send and receive message (can simulate both Synchronous and Asynchronous). This class creates the ClientBootstrap, set the pipeline factory, invoke the bootstrap.connect() and eventually get a handle/reference of the class A through which to be used to send and receive message. Something like:
ChannelFuture future = bootstrap.connect();
Channel channel = future.awaitUninterruptibly().getChannel();
A handler = channel.getPipeline().get(A.class);
I know in class A, I can override
public void channelClosed(ChannelHandlerContext ctx, ChannelStateEvent e);
so that when the remote server is down, I can be notified.
Since after channel is closed, the original class A reference (handler above) in class B is not valid anymore, so I need to replace it with a new reference.
Ideally, I want class A to have a mechanism to notify class B within the above overrided channelClosed method so the bootstrap.connect can be invoked again within class B. One way to do this is to have a reference in class A that reference class B. To do that, I would need to pass class B reference to the PipelineFactory and then have the PipelineFactory pass the reference of B to A.
Any other simpler way to achieve the same thing?
thanks,
Channel.closeFuture() returns a ChannelFuture that will notify you when the channel is closed. You can add a ChannelFutureListener to the future in B so that you can make another connection attempt there.
You probably want to repeat this until the connection attempt succeeds finally:
private void doConnect() {
Bootstrap b = ...;
b.connect().addListener((ChannelFuture f) -> {
if (!f.isSuccess()) {
long nextRetryDelay = nextRetryDelay(...);
f.channel().eventLoop().schedule(nextRetryDelay, ..., () -> {
doConnect();
}); // or you can give up at some point by just doing nothing.
}
});
}
I don't know if this is the right solution but to fix the thread leak of trustin's solution I found I could shutdown the event loop after the scheduler had triggered:
final EventLoop eventloop = f.channel().eventLoop();
b.connect().addListener((ChannelFuture f) -> {
if (!f.isSuccess()) {
long nextRetryDelay = nextRetryDelay(...);
eventloop.schedule(() -> {
doConnect();
eventloop.shutdownGracefully();
}, nextRetryDelay, ...);
}
});
Here's another version encapsulating the reconnect behavior in a small helper class
Bootstrap clientBootstrap...
EventLoopGroup group = new NioEventLoopGroup();
Session session = new Session(clientBootstrap,group);
Disposable shutdownHook = session.start();
interface Disposable {
void dispose();
}
class Session implements Disposable{
private final EventLoopGroup scheduler;
private final Bootstrap clientBootstrap;
private int reconnectDelayMs;
private Channel activeChannel;
private AtomicBoolean shouldReconnect;
private Session(Bootstrap clientBootstrap, EventLoopGroup scheduler) {
this.scheduler = scheduler;
this.clientBootstrap = clientBootstrap;
this.reconnectDelayMs = 1;
this.shouldReconnect = new AtomicBoolean(true);
}
public Disposable start(){
//Create a new connectFuture
ChannelFuture connectFuture = clientBootstrap.connect();
connectFuture.addListeners( (ChannelFuture cf)->{
if(cf.isSuccess()){
L.info("Connection established");
reconnectDelayMs =1;
activeChannel = cf.channel();
//Listen to the channel closing
var closeFuture =activeChannel.closeFuture();
closeFuture.addListeners( (ChannelFuture closeFut)->{
if(shouldReconnect.get()) {
activeChannel.eventLoop().schedule(this::start, nextReconnectDelay(), TimeUnit.MILLISECONDS);
}
else{
L.info("Session has been disposed won't reconnect");
}
});
}
else{
int delay =nextReconnectDelay();
L.info("Connection failed will re-attempt in {} ms",delay);
cf.channel().eventLoop().schedule(this::start,delay , TimeUnit.MILLISECONDS);
}
});
return this;
}
/**
* Call this to end the session
*/
#Override
public void dispose() {
try {
shouldReconnect.set(false);
scheduler.shutdownGracefully().sync();
if(activeChannel !=null) {
activeChannel.closeFuture().sync();
}
}catch(InterruptedException e){
L.warn("Interrupted while shutting down TcpClient");
}
}
private int nextReconnectDelay(){
this.reconnectDelayMs = this.reconnectDelayMs*2;
return Math.min(this.reconnectDelayMs, 5000);
}
}
Related
I have the following setup. There is a message distributor that spreads inbound client messages across a configured number of message queues (LinkedBlockingQueues in my case), based on an unique identifier called appId (per connected client):
public class MessageDistributor {
private final List<BlockingQueue<MessageWrapper>> messageQueueBuckets;
public MessageDistributor(List<BlockingQueue<MessageWrapper>> messageQueueBuckets) {
this.messageQueueBuckets = messageQueueBuckets;
}
public void handle(String appId, MessageWrapper message) {
int index = (messageQueueBuckets.size() - 1) % hash(appId);
try {
messageQueueBuckets.get(index).offer(message);
} catch (Exception e) {
// handle exception
}
}
}
As I also need to answer the message later on, I wrap the message object and the netty channel inside a MessageWrapper:
public class MessageWrapper {
private final Channel channel;
private final Message message;
public MessageWrapper(Channel channel, Message message) {
this.channel = channel;
this.message = message;
}
public Channel getChannel() {
return channel;
}
public Message getMessage() {
return message;
}
}
Furthermore, there is a message consumer, which implements a Runnable and takes new messages from the assigned blocking queue. This guy performs some expensive/blocking operations that I want to have outside the main netty event loop and which should also not block operations for other connected clients too much, hence the usage of several queues:
public class MessageConsumer implements Runnable {
private final BlockingQueue<MessageWrapper> messageQueue;
public MessageConsumer(BlockingQueue<MessageWrapper> messageQueue) {
this.messageQueue = messageQueue;
}
#Override
public void run() {
while (true) {
try {
MessageWrapper msgWrap = messageQueue.take();
Channel channel = msgWrap.getChannel();
Message msg = msgWrap.getMessage();
doSthExepnsiveOrBlocking(channel, msg);
} catch (Exception e) {
// handle exception
}
}
}
public void doSthExepnsiveOrBlocking(Channel channel, Message msg) {
// some expsive/blocking operations
channe.writeAndFlush(someResultObj);
}
}
The setup of all classes looks like the following (the messageExecutor is a DefaultEventeExecutorGroup with a size of 8):
int nrOfWorkers = config.getNumberOfClientMessageQueues();
List<BlockingQueue<MessageWrapper>> messageQueueBuckets = new ArrayList<>(nrOfWorkers);
for (int i = 0; i < nrOfWorkers; i++) {
messageQueueBuckets.add(new LinkedBlockingQueue<>());
}
MessageDistributor distributor = new MessageDistributor(messageQueueBuckets);
List<MessageConsumer> consumers = new ArrayList<>(nrOfWorkers);
for (BlockingQueue<MessageWrapper> messageQueueBucket : messageQueueBuckets) {
MessageConsumer consumer = new MessageConsumer(messageQueueBucket);
consumers.add(consumer);
messageExecutor.submit(consumer);
}
My goal with this approach is to isolate connected clients from each other (not fully, but at least a bit) and also to execute expensive operations on different threads.
Now my question is: Is it valid to wrap the netty channel object inside this MessageWrapper for later use and access its write method in some other thread?
UPDATE
Instead of building additional message distribution mechanics on top of netty, I decided to simply go with a separate EventExecutorGroup for my blocking channel handlers and see how it works.
Yes it is valid to call Channel.* methods from other threads. That said the methods perform best when these are called from the EventLoop thread itself that belongs to the Channel.
I'm trying to implement a UDP server with Netty. The idea is to bind only once (therefore creating only one Channel). This Channel is initialized with only one handler that dispatches processing of incoming datagrams among multiple threads via an ExecutorService.
#Configuration
public class SpringConfig {
#Autowired
private Dispatcher dispatcher;
private String host;
private int port;
#Bean
public Bootstrap bootstrap() throws Exception {
Bootstrap bootstrap = new Bootstrap()
.group(new NioEventLoopGroup(1))
.channel(NioDatagramChannel.class)
.option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT)
.handler(dispatcher);
ChannelFuture future = bootstrap.bind(host, port).await();
if(!future.isSuccess())
throw new Exception(String.format("Fail to bind on [host = %s , port = %d].", host, port), future.cause());
return bootstrap;
}
}
#Component
#Sharable
public class Dispatcher extends ChannelInboundHandlerAdapter implements InitializingBean {
private int workerThreads;
private ExecutorService executorService;
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
DatagramPacket packet = (DatagramPacket) msg;
final Channel channel = ctx.channel();
executorService.execute(new Runnable() {
#Override
public void run() {
//Process the packet and produce a response packet (below)
DatagramPacket responsePacket = ...;
ChannelFuture future;
try {
future = channel.writeAndFlush(responsePacket).await();
} catch (InterruptedException e) {
return;
}
if(!future.isSuccess())
log.warn("Failed to write response packet.");
}
});
}
#Override
public void afterPropertiesSet() throws Exception {
executorService = Executors.newFixedThreadPool(workerThreads);
}
}
I have the following questions:
Should the DatagramPacket received by the channelRead method of the Dispatcher class be duplicated before being used by the worker thread? I wonder if this packet is destroyed after the channelRead method returns, even if a reference is kept by the worker thread.
Is it safe to share the Channel among all the worker threads and let them call writeAndFlush concurrently?
Thanks!
Nope. If you need the object to live longer you either turn it into something else or use ReferenceCountUtil.retain(datagram) and then ReferenceCountUtil.release(datagram) once you're done with it. You also shouldn't be doing await() at the executor service as well, you should register a handler for whatever happens.
Yes, channel objects are thread safe and they can be called from many different threads.
This is the code that works. It send a message to an Actor (Greeter) and wait for the answer back.
But it blocks the current thread.
public class Future1Blocking {
public static void main(String[] args) throws Exception {
ActorSystem system = ActorSystem.create("system");
final ActorRef actorRef = system.actorOf(Props.create(Greeter.class), "greeter");
Timeout timeout = new Timeout(Duration.create(5, "seconds"));
Future<Object> future = Patterns.ask(actorRef, Greeter.Msg.GREET, timeout);
// this blocks current running thread
Greeter.Msg result = (Greeter.Msg) Await.result(future, timeout.duration());
System.out.println(result);
}
}
What is the possible way for my example to use future.onSuccess to get the result without blocking the current calling thread?
Ahh. that's was easy (sorry).
future.onSuccess(new PrintResult<Object>(), system.dispatcher());
Where:
public final class PrintResult<T> extends OnSuccess<T> {
#Override public final void onSuccess(T t) {
System.out.println(t);
}
}
I am using Netty 3.2.7. I am trying to write functionality in my client such that if no messages are written after a certain amount of time (say, 30 seconds), a "keep-alive" message is sent to the server.
After some digging, I found that WriteTimeoutHandler should enable me to do this. I found this explanation here: https://issues.jboss.org/browse/NETTY-79.
The example given in the Netty documentation is:
public ChannelPipeline getPipeline() {
// An example configuration that implements 30-second write timeout:
return Channels.pipeline(
new WriteTimeoutHandler(timer, 30), // timer must be shared.
new MyHandler());
}
In my test client, I have done just this. In MyHandler, I also overrided the exceptionCaught() method:
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) {
if (e.getCause() instanceof WriteTimeoutException) {
log.info("Client sending keep alive!");
ChannelBuffer keepAlive = ChannelBuffers.buffer(KEEP_ALIVE_MSG_STR.length());
keepAlive.writeBytes(KEEP_ALIVE_MSG_STR.getBytes());
Channels.write(ctx, Channels.future(e.getChannel()), keepAlive);
}
}
No matter what duration the client does not write anything to the channel, the exceptionCaught() method I have overridden is never called.
Looking at the source of WriteTimeoutHandler, its writeRequested() implementation is:
public void writeRequested(ChannelHandlerContext ctx, MessageEvent e)
throws Exception {
long timeoutMillis = getTimeoutMillis(e);
if (timeoutMillis > 0) {
// Set timeout only when getTimeoutMillis() returns a positive value.
ChannelFuture future = e.getFuture();
final Timeout timeout = timer.newTimeout(
new WriteTimeoutTask(ctx, future),
timeoutMillis, TimeUnit.MILLISECONDS);
future.addListener(new TimeoutCanceller(timeout));
}
super.writeRequested(ctx, e);
}
Here, it seems that this implementation says, "When a write is requested, make a new timeout. When the write succeeds, cancel the timeout."
Using a debugger, it does seem that this is what is happening. As soon as the write completes, the timeout is cancelled. This is not the behavior I want. The behavior I want is: "If the client has not written any information to the channel for 30 seconds, throw a WriteTimeoutException."
So, is this not what WriteTimeoutHandler is for? This is how I interpreted it from what I've read online, but the implementation does not seem to work this way. Am I using it wrong? Should I use something else? In our Mina version of the same client I am trying to rewrite, I see that the sessionIdle() method is overridden to achieve the behavior I want, but this method is not available in Netty.
For Netty 4.0 and newer, you should extend ChannelDuplexHandler like in example from IdleStateHandler documentation :
// An example that sends a ping message when there is no outbound traffic
// for 30 seconds. The connection is closed when there is no inbound traffic
// for 60 seconds.
public class MyChannelInitializer extends ChannelInitializer<Channel> {
#Override
public void initChannel(Channel channel) {
channel.pipeline().addLast("idleStateHandler", new IdleStateHandler(60, 30, 0));
channel.pipeline().addLast("myHandler", new MyHandler());
}
}
// Handler should handle the IdleStateEvent triggered by IdleStateHandler.
public class MyHandler extends ChannelDuplexHandler {
#Override
public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
if (evt instanceof IdleStateEvent) {
IdleStateEvent e = (IdleStateEvent) evt;
if (e.state() == IdleState.READER_IDLE) {
ctx.close();
} else if (e.state() == IdleState.WRITER_IDLE) {
ctx.writeAndFlush(new PingMessage());
}
}
}
}
I would suggest to add the IdleStateHandler and then add your custom implementation of IdleStateAwareUpstreamHandler which can react on the idle state. This works out very well for me on many different projects.
The javadocs list the following example, that you could use as the base of your implementation:
public class MyPipelineFactory implements ChannelPipelineFactory {
private final Timer timer;
private final ChannelHandler idleStateHandler;
public MyPipelineFactory(Timer timer) {
this.timer = timer;
this.idleStateHandler = new IdleStateHandler(timer, 60, 30, 0);
// timer must be shared.
}
public ChannelPipeline getPipeline() {
return Channels.pipeline(
idleStateHandler,
new MyHandler());
}
}
// Handler should handle the IdleStateEvent triggered by IdleStateHandler.
public class MyHandler extends IdleStateAwareChannelHandler {
#Override
public void channelIdle(ChannelHandlerContext ctx, IdleStateEvent e) {
if (e.getState() == IdleState.READER_IDLE) {
e.getChannel().close();
} else if (e.getState() == IdleState.WRITER_IDLE) {
e.getChannel().write(new PingMessage());
}
}
}
ServerBootstrap bootstrap = ...;
Timer timer = new HashedWheelTimer();
...
bootstrap.setPipelineFactory(new MyPipelineFactory(timer));
...
Motivation
I want extra eyes to confirm that I am able to call this method XMPPConnection.sendPacket(
Packet ) concurrently. For my current code, I am invoking a List of Callables (max 3) in a serial fashion. Each Callable sends/receives XMPP packets on the one piece of XMPPConnection. I plan to parallelize these Callables by spinning off multiple threads & each Callable will invoke sendPacket on the shared XMPPConnection without synchronization.
XMPPConnection
class XMPPConnection
{
private boolean connected = false;
public boolean isConnected()
{
return connected;
}
PacketWriter packetWriter;
public void sendPacket( Packet packet )
{
if (!isConnected())
throw new IllegalStateException("Not connected to server.");
if (packet == null)
throw new NullPointerException("Packet is null.");
packetWriter.sendPacket(packet);
}
}
PacketWriter
class PacketWriter
{
public void sendPacket(Packet packet)
{
if (!done) {
// Invoke interceptors for the new packet
// that is about to be sent. Interceptors
// may modify the content of the packet.
processInterceptors(packet);
try {
queue.put(packet);
}
catch (InterruptedException ie) {
ie.printStackTrace();
return;
}
synchronized (queue) {
queue.notifyAll();
}
// Process packet writer listeners. Note that we're
// using the sending thread so it's expected that
// listeners are fast.
processListeners(packet);
}
protected PacketWriter( XMPPConnection connection )
{
this.queue = new ArrayBlockingQueue<Packet>(500, true);
this.connection = connection;
init();
}
}
What I conclude
Since the PacketWriter is using a BlockingQueue, there is no problem with my intention to invoke sendPacket from multiple threads. Am I correct ?
Yes you can send packets from different threads without any problems.
The Smack blocking queue is because what you can't do is let the different threads write the output stream at the same time. Smack takes the responsibility of synchronizing the output stream by writing it with a per packet granularity.
The pattern implemented by Smack is simply a typical producer/consumer concurrency pattern. You may have several producers (your threads) and only one consumer (the Smack's PacketWriter running in it's own thread).
Regards.
You haven't provided enough information here.
We don't know how the following are implemented:
processInterceptors
processListeners
Who reads / writes the 'done' variable? If one thread sets it to true, then all the other threads will silently fail.
From a quick glance, this doesn't look thread safe, but there's no way to tell for sure from what you've posted.
Other issues:
Why is PacketWriter a class member of XMPPConnectionwhen it's only used in one method?
Why does PacketWriter have a XMPPConnection member var and not use it?
You might consider using a BlockingQueue if you can restrict to Java 5+.
From the Java API docs, with a minor change to use ArrayBlockingQueue:
class Producer implements Runnable {
private final BlockingQueue queue;
Producer(BlockingQueue q) { queue = q; }
public void run() {
try {
while(true) { queue.put(produce()); }
} catch (InterruptedException ex) { ... handle ...}
}
Object produce() { ... }
}
class Consumer implements Runnable {
private final BlockingQueue queue;
Consumer(BlockingQueue q) { queue = q; }
public void run() {
try {
while(true) { consume(queue.take()); }
} catch (InterruptedException ex) { ... handle ...}
}
void consume(Object x) { ... }
}
class Setup {
void main() {
BlockingQueue q = new ArrayBlockingQueue();
Producer p = new Producer(q);
Consumer c1 = new Consumer(q);
Consumer c2 = new Consumer(q);
new Thread(p).start();
new Thread(c1).start();
new Thread(c2).start();
}
}
For your usage you'd have your real sender (holder of the actual connection) be the Consumer, and packet preparers/senders be the producers.
An interesting additional thought is that you could use a PriorityBlockingQueue to allow flash override XMPP packets that are sent before any other waiting packets.
Also, Glen's points on the design are good points. You might want to take a look at the Smack API (http://www.igniterealtime.org/projects/smack/) rather than creating your own.