Develop Reference

SSL Handshake is not happening from the pipeline while doing reconnection to servers

Problem I'm facing is the something like this :
in Main class, I'll try to connect to a server and attach Channel Listener for future actions.
If Connection establishes successfully, SSL Handshake is done without any problem.
But if Connection in Step 1 fails, I'll try to connect to same or different server and again attach same channel listener same as point.
But expectation is it should SSL handshake as before in point 2 if connection is established. But it's not. Even if I forcefully call renegotiate metthod in SslHandler.
Expected behavior
If any connection exception using bootstrap object to connect to the server, expectation is it should SSL handshake.
Actual behavior
It's skipping the SSL handshake while retrying and failing with UnknownMessage type expected(ByteBuf)
Steps to reproduce
While Main Connection
public class Main {
private final static Logger LOGGER = LoggerFactory.getLogger(Main.class);
public static void main(String[] args) {
ClientConfig clientConfig = null;
LOGGER.info("initializing Agent Stats uploader");
// Set up.
InternalLoggerFactory.setDefaultFactory(Slf4JLoggerFactory.INSTANCE);
Bootstrap clientBootstrap = getBootstrap();
clientConfig = ClientConfig.getInstance();
InetSocketAddress server = clientConfig.getPrimaryScnInetAddrs();
Objects.nonNull(server.getHostName());
Objects.nonNull(server.getPort());
// Make a new connection.
LOGGER.info("Initialization complete, ready to connect to the host and port {}:{}", server.getHostName(),
server.getPort());
ServerChannelFutureListener serverChannelFutureListener = ServerChannelFutureListener.getInstance();
serverChannelFutureListener.setClientBootStrap(clientBootstrap);
ChannelPromise channelPromise =
(ChannelPromise) clientBootstrap.connect(server).addListener(serverChannelFutureListener);
EventLoopGroup eventGroupExecutor = clientBootstrap.config().group();
AgentStatsProcess agentStatsThread = AgentStatsProcess.getInstance();
agentStatsThread.setParentChannelFuture(channelPromise);
eventGroupExecutor.scheduleAtFixedRate(agentStatsThread, clientConfig.getInitialDelay(),
clientConfig.getScheduleInterval(), TimeUnit.SECONDS);
LOGGER.info("Scheduled Agent Stats uploading, should start in 30 secs");
LOGGER.info("Connection complete");
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
LOGGER.info("Killing AgentStatUploader Thread");
eventGroupExecutor.shutdownGracefully();
}));
}
public static final Bootstrap getBootstrap() {
EventLoopGroup group = new NioEventLoopGroup();
Bootstrap b = new Bootstrap();
b.group(group);
b.channel(NioSocketChannel.class);
b.handler(new AgentStatsChannelInitializationHandler());
b.option(ChannelOption.SO_KEEPALIVE, true);
b.option(ChannelOption.TCP_NODELAY, true);
return b;
}
}
Having Channel Future handler for implementing re-try logic in step 1
public final class ServerChannelFutureListener implements GenericFutureListener {
private static final Logger logger = LoggerFactory.getLogger(ServerChannelFutureListener.class.getName());
private static ServerChannelFutureListener instance;
private AtomicInteger count = new AtomicInteger(1);
private ClientConfig clientConfig = ClientConfig.getInstance();
private boolean isPrimary=true;
private ChannelFuture channelFuture;
private Bootstrap clientBootStrap;
private long timeout;
private ServerChannelFutureListener(){
this.timeout = clientConfig.getRetryAfter();
}
#override
public void operationComplete(ChannelFuture future) throws Exception {
channelFuture = future;
int maxretries = clientConfig.getMaxRetries();
if (!future.isSuccess()) {
logger.info("Connection to {} scn is not successful, retrying ({}/{})", getServerType(), count.get(),maxretries);
logger.debug("Connection to server is failed with error: ",future.cause());
if ( count.incrementAndGet() > maxretries) {
// fails to connect even after max-retries, try to connect to next server.
logger.info("Failed to connect to {} server, will try to connect to {} now.",
getServerType(),
isPrimary() ? "SECONDARY":"PRIMARY");
count.getAndSet(1);
isPrimary = !isPrimary();
this.timeout = clientConfig.getRetryAfter();
logger.info("Connecting Server type changed, so resetting timeout: {}", this.timeout);
}else{
// retry
logger.info("Exponential Back-off set to: {} secs, waiting for next server connection", this.timeout);
//TimeUnit.SECONDS.sleep(this.timeout);
this.timeout = ExpontentialBackOff.getNextBackOff(this.timeout);
}
InetSocketAddress server = getServer();
logger.info("Initialization complete, ready to connect to the host and port {}:{}", server.getHostName(),
server.getPort());
channelFuture = clientBootStrap.connect(server).addListener(this);
}else {
logger.info("Using Connection with config: {}, to Server {} ", future.channel().config(),
future.channel().localAddress());
this.timeout = clientConfig.getRetryAfter();
logger.info("Time out Back-off reset to: {} for next server connection", this.timeout);
}
AgentStatsProcess.getInstance().setParentChannelFuture(channelFuture);
}
private String getServerType() {
return isPrimary() ? "PRIMARY" : "SECONDARY";
}
private InetSocketAddress getServer(){
return isPrimary()?clientConfig.getPrimaryScnInetAddrs():clientConfig.getSecondaryScnInetAddrs();
}
public static ServerChannelFutureListener getInstance(){
if(null == instance){
instance = new ServerChannelFutureListener();
}
return instance;
}
public boolean isPrimary() {
return isPrimary;
}
public ChannelFuture getChannelFuture() {
return channelFuture;
}
public void setClientBootStrap(Bootstrap cb) {
this.clientBootStrap = cb;
}
}
Expectation is SSL Handshake should happen after trying to reconnect but its failing.
Netty version: 4.1.12.Final

Fixed this issue, Culprit here is "ProtobufVarint32FrameDecoder " and it's parent Class "ByteToMessageDecoder". "ByteToMessageDecoder" make sure it's child classes are not shareable.
Because above classes are not shareable, every time code try to reconnect using boostrap, initializer class fails to add handlers in pipeline results in "ctx.close()" and no handlers.
I've did work-around of adding those two classes into my project and raised #10371 bug to address this issue.

Handle UnknownHostException properly in RabbitMQ, preventing auto recovery

I have written a Spring Application that is implementing RabbitMQ manually with the RabbitMQ Client API.
The way the Connection Factory and Connection are set are similar to the tutorial:
public class Recv {
private final static String QUEUE_NAME = "hello";
public static void main(String[] argv)
throws java.io.IOException,
java.lang.InterruptedException {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("122.34.1.1");
factory.setPort(5672);
factory.setUsername("user");
factory.setPassword("password")
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
System.out.println(" [*] Waiting for messages. To exit press CTRL+C");
...
}
}
The connection works correct. However, if I turn off my host server, my application seems to stuck in some sort of loop where it tries to auto recover by keep pinging the turned-off host.
Because of this, my console get filled with tons of stacktraces that say "UnknownHostException". The exact location according to stacktraces is the line:
Connection connection = factory.newConnection();
I have tried to put a try-catch block around this line, but that doesn't seem to work at all.
If the traditional try-catch block can't handle the exception coming from the connection, what is the proper way to catch the exception and stop the auto-recovery from creating this loop?
Thanks.

Try setting up your Rabbit like this (on both ends):
#Bean
public ConnectionFactory connectionFactory() {
final CachingConnectionFactory connectionFactory = new CachingConnectionFactory(server);
connectionFactory.setUsername("user");
connectionFactory.setPassword("pass");
return connectionFactory;
}
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory() {
final SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory());
return factory;
}
The port should be set by default. If not, it will show on startup and you can change it. The Queue should be declared and set by the consumer (host or whatever you call it).
And put a #EnableRabbit annotation on your configuration class if you haven't.
Consumer declares a Queue:
#RabbitListener(queues = "myQueue")
#Component
public class RabbitListener {
#Bean
public Queue getQueue() {
return new Queue("myQueue");
}
#RabbitHandler
public void getElementFromMyQueue(#Payload Object object) {
// handle object as you want
}
}
After that just #Autowire the RabbitTemplate on the sender and you should be good to go. The queue should remain idle but 'active', even when the host is down

Active MQ connection issue

Hi am working on wso2 esb and using Active MQ for message queue.
I have a simple service to place a message in which it call custom java class where it creates a tcp connection and drops a message in queue.
Java code looks like below
package in.esb.custommediators;
import javax.jms.*;
import org.apache.activemq.ActiveMQConnection;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.synapse.ManagedLifecycle;
import org.apache.synapse.MessageContext;
import org.apache.synapse.core.SynapseEnvironment;
import org.apache.synapse.mediators.AbstractMediator;
import org.apache.synapse.core.axis2.Axis2MessageContext;
import org.apache.synapse.transport.nhttp.NhttpConstants;
import org.json.JSONObject;
import org.json.XML;
public class JMSStoreMediator extends AbstractMediator implements
ManagedLifecycle {
Connection connection;
Session session;
public boolean mediate(MessageContext msgCtx) {
log.info("LogLocation = "+getClass().getName()+",ProxyName = "+msgCtx.getProperty("proxy.name")+
",Usercode = "+msgCtx.getProperty("usercode")+",Clientid = "+msgCtx.getProperty("clientid")+
",requestMsgId = "+msgCtx.getProperty("requestMsgId")+",Position = START");
try {
boolean topic=false;
String jmsuri=""+msgCtx.getProperty("jmsuri");
String t=""+msgCtx.getProperty("topic");
if(t.isEmpty()){
topic=false;
}
else {
topic=Boolean.valueOf(t);
}
ConnectionFactory factory= new ActiveMQConnectionFactory(jmsuri);
connection = factory.createConnection();
connection.start();
log.info("LogLocation = "+getClass().getName()+",JMS connection created :"+connection);
this.session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Destination destination=null;
if(!topic)destination= session.createQueue(""+msgCtx.getProperty("jmsqueue"));
else destination= session.createTopic(""+msgCtx.getProperty("jmsqueue"));
MessageProducer producer = session.createProducer(destination);
producer.setDeliveryMode(DeliveryMode.PERSISTENT);
String xml = ""+msgCtx.getEnvelope().getBody().toStringWithConsume();
if(topic){
JSONObject obj=XML.toJSONObject(xml);
JSONObject ar=obj.getJSONObject("soapenv:Body");
ar.remove("xmlns:soapenv");
xml=ar.toString();
}
TextMessage message = session.createTextMessage(xml);
producer.send(message);
} catch (Exception e) {
log.info("LogLocation = "+getClass().getName()+",Error in storing message in JMS stacktrace is :"+e.toString()+"message is :"+e.getMessage());
e.printStackTrace();
((Axis2MessageContext) msgCtx).setProperty(NhttpConstants.HTTP_SC, 500);
handleException("Error while storing in the message store", msgCtx);
}
finally {
try {
session.close();
if (connection!=null){
log.info("LogLocation = "+getClass().getName()+",JMS connection closing :"+connection);
connection.close();
}
} catch (JMSException e) {
log.info("LogLocation = "+getClass().getName()+",Error in closing JMS connection stacktrace is :"+e.toString());
e.printStackTrace();
}
}
return true;
}
#Override
public void destroy() {
// TODO Auto-generated method stub
}
#Override
public void init(SynapseEnvironment arg0) {
// TODO Auto-generated method stub
}
}
when i call this service to send a message in queue below logs get generated.
[2017-07-29 11:18:35,962] INFO - JMSStoreMediator LogLocation = in.esb.custommediators.JMSStoreMediator,JMS connection created :ActiveMQConnection {id=ID:my-desktop-36442-1501307315570-3:1,clientId=ID:my-desktop-36442-1501307315570-2:1,started=true}
As of now every thing is working good , But when two users try to submit message at the same tire some strange thing happen as shown below
[2017-07-29 11:43:11,948] INFO - JMSStoreMediator LogLocation = in.my.esb.custommediators.JMSStoreMediator,JMS connection created :ActiveMQConnection {id=ID:my-desktop-36442-1501307315570-11:1,clientId=ID:my-desktop-36442-1501307315570-10:1,started=false}
[2017-07-29 11:43:11,963] INFO - JMSStoreMediator LogLocation = in.my.esb.custommediators.JMSStoreMediator,JMS connection created :ActiveMQConnection {id=ID:my-desktop-36442-1501307315570-11:1,clientId=ID:my-desktop-36442-1501307315570-10:1,started=true}
[2017-07-29 11:43:12,068] INFO - JMSStoreMediator LogLocation = in.my.esb.custommediators.JMSStoreMediator,Error in closing JMS connection stacktrace is :org.apache.activemq.ConnectionClosedException: The connection is already closed
Active MQ is creating two connections but using one connection for both the calls and that one connection is getting closed in one of the service call and throwing already closed error in the other service call and the other connection is waiting forever in the connection list of active mq with active status true as shown in the below image and this is also seen in the ESB thread list.
This kind of connections pileup and causing hangs ESB server. Even if i reset this connections from Active MQ ESB threads carry this connection info and only after a restart of ESB the problem get fixed.

Have you read article Extending the Functionality of WSO2 Enterprise Service Bus - Part 1 ?
Important part is Threading Safety. It states, each mediator, including custom, is shared between incoming messages. I recommend to move class variables
Connection connection;
Session session;
to method public boolean mediate(MessageContext msgCtx) since local variables are thread safe
public class JMSStoreMediator extends AbstractMediator implements
ManagedLifecycle {
public boolean mediate(MessageContext msgCtx) {
Connection connection;
Session session;
....
....
rest the same

How to get notified when a producer's connection dropped?

I am using an embedded ActiveMQ broker.
My goal is to find a way to detect when an external producer on a Queue lost it's connection.
I am starting the broker like this:
BrokerService broker = new BrokerService();
broker.addConnector("tcp://" + LISTEN_DEVICE_IP + ":" + port);
setLastMessagesPersistent(broker);
broker.start();
ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory("vm://localhost");
connection = factory.createConnection();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
connection.start();
And after that I tried to add a TransportListener:
((ActiveMQConnection) connection).addTransportListener(new TransportListener() {
public void transportResumed() {
System.out.println("resumed");
}
public void transportInterupted() {
System.out.println("interrupted");
}
public void onException(IOException arg0) {
System.out.println("ioexception: " + arg0);
}
public void onCommand(Object arg0) {
System.out.println("command: " + arg0);
}
});
I also register a consumer and ProducerListener like this:
Destination dest = session.createQueue(queuename);
MessageConsumer consumer = session.createConsumer(dest);
ProducerEventSource source = new ProducerEventSource(connection, dest);
System.out.println("Setting Producer Listener");
source.setProducerListener(prodevent -> {
System.out.println("producer status: " + prodevent.isStarted());
});
// Gets called from inside the broker's Thread and somehow causes deadlocks if I don't invoke this from the outside
new Thread(() -> {
try {
consumer.setMessageListener(new NetworkEventPlayerAdapter(objectMapper, event, gameEventManager, playerID));
} catch (Exception e) {
e.printStackTrace();
}
}).start();
Unfortunately neither the TransportListener, nor the ProducerListener give me any output when I forcefully quit another application that previously was added as a producer (Alt+F4). The broker surely notices though:
WARN | Transport Connection to: tcp://127.0.0.1:58988 failed: java.net.SocketException: Connection reset
WARN | Transport Connection to: tcp://127.0.0.1:58986 failed: java.net.SocketException: Connection reset
But I don't find a way to get a callback on those events in Java.
I also tried setting a custom IOExceptionHandler in the broker and adding an ExceptionListener to the connection. They also get never called.

You can use The Advisory Topics ActiveMQ.Advisory.Connection or even the ActiveMQ.Advisory.Producer.Queue ActiveMQ.Advisory.Producer.Topic , these provide the stats on connections of number of producers , check this link http://activemq.apache.org/advisory-message.html

a possible way is to parse the log-output for Connection reset
WARN | Transport Connection to: tcp://127.0.0.1:58988 failed:
java.net.SocketException: Connection reset
WARN | Transport
Connection to: tcp://127.0.0.1:58986 failed: java.net.SocketException:
Connection reset
Because the socket is implemented in activemq you would have to add the ExceptionListener there to write your own exception handling routine ...

ActiveMQ connection times out even when wireFormat.maxInactivityDuration=0

I am listening to a queue of activemq.
My configuration is like below:
ACTIVEMQ_BROKER_URL =failover:(tcp://LON6QAEQTAPP01:61616?wireFormat.maxInactivityDuration=0)
From the console logs I can see that a reconnect attempt is made to connect to this server.
[2014-08-20 08:57:43,303] INFO 236[ActiveMQ Task-1] -
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1030)
- Successfully connected to tcp://LON6QAEQTAPP01:61616?wireFormat.maxInactivityDuration=0
[2014-08-20 08:57:43,355] INFO 288[ActiveMQ Task-1] -
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1030)
- Successfully connected to tcp://LON6QAEQTAPP01:61616?wireFormat.maxInactivityDuration=0
[2014-08-20 08:57:43,374] INFO 307[ActiveMQ Task-1] -
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1030)
- Successfully connected to tcp://LON6QAEQTAPP01:61616?wireFormat.maxInactivityDuration=0
Still I am not able to consume the message. I am using the following code to try and handle it from code side but still not able to get the connection persistent.
try
{
connection.start();
while (true)
{
try
{
if (connection.isClosed() || connection.isTransportFailed() || session.isClosed() || !connection.getTransport().isConnected())
{
log.info("Connection was reset, re-connecting..");
connection.cleanup();
init();
connection.start();
}
}
catch (Throwable t)
{
init();
log.error("Connection was reset, re-connecting in exception block", t);
}
Thread.sleep(30000L);
}
private void init() throws Exception
{
init(brokerURL, queueName, userName, password, manual);
}
public void init(String brokerURL, String queueName, String userName, String password, boolean manual) throws Exception
{
this.manual = manual;
this.brokerURL = brokerURL;
this.queueName = queueName;
this.userName = userName;
this.password = password;
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(userName, password, brokerURL);
connection = (ActiveMQConnection) connectionFactory.createConnection();
connection.addTransportListener(new TransportListener()
{
#Override
public void transportResumed()
{
log.info("Transport Resumed ");
}
#Override
public void transportInterupted()
{
log.info("Transport Interupted ");
}
#Override
public void onException(IOException arg0)
{
arg0.printStackTrace();
log.error("Transport Exception: " + arg0.getMessage());
}
#Override
public void onCommand(Object arg0)
{
// TODO Auto-generated method stub
}
});
connection.setExceptionListener(this);
connection.setCloseTimeout(10);
session = (ActiveMQSession) connection.createSession(false, Session.AUTO_ACKNOWLEDGE);// CLIENT_ACKNOWLEDGE
Destination destination = session.createQueue(queueName);
MessageConsumer consumer = session.createConsumer(destination);
consumer.setMessageListener(this);
}
Can someone help me as to how can I get this connection to be active alive all the time. It generally times out after inactivity of 40 mins or so.

Trying to force reconnect an open connection the way you are is definitely not going to do you and good. You need to tear down the connection and create a new one if you want to handle connection interruption yourself. What you really should do is figure out what is closing your connection on you, probably a firewall closing down the inactive connection or a load balancer. Look at your environment and see what else is in the mix that could be closing the connection on you.
The broker can, if configured to do so, rebalance clients in a cluster. Are you using multiple clients in a clustered environment etc?