Hi am working on wso2 esb and using Active MQ for message queue.
I have a simple service to place a message in which it call custom java class where it creates a tcp connection and drops a message in queue.
Java code looks like below
package in.esb.custommediators;
import javax.jms.*;
import org.apache.activemq.ActiveMQConnection;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.synapse.ManagedLifecycle;
import org.apache.synapse.MessageContext;
import org.apache.synapse.core.SynapseEnvironment;
import org.apache.synapse.mediators.AbstractMediator;
import org.apache.synapse.core.axis2.Axis2MessageContext;
import org.apache.synapse.transport.nhttp.NhttpConstants;
import org.json.JSONObject;
import org.json.XML;
public class JMSStoreMediator extends AbstractMediator implements
ManagedLifecycle {
Connection connection;
Session session;
public boolean mediate(MessageContext msgCtx) {
log.info("LogLocation = "+getClass().getName()+",ProxyName = "+msgCtx.getProperty("proxy.name")+
",Usercode = "+msgCtx.getProperty("usercode")+",Clientid = "+msgCtx.getProperty("clientid")+
",requestMsgId = "+msgCtx.getProperty("requestMsgId")+",Position = START");
try {
boolean topic=false;
String jmsuri=""+msgCtx.getProperty("jmsuri");
String t=""+msgCtx.getProperty("topic");
if(t.isEmpty()){
topic=false;
}
else {
topic=Boolean.valueOf(t);
}
ConnectionFactory factory= new ActiveMQConnectionFactory(jmsuri);
connection = factory.createConnection();
connection.start();
log.info("LogLocation = "+getClass().getName()+",JMS connection created :"+connection);
this.session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Destination destination=null;
if(!topic)destination= session.createQueue(""+msgCtx.getProperty("jmsqueue"));
else destination= session.createTopic(""+msgCtx.getProperty("jmsqueue"));
MessageProducer producer = session.createProducer(destination);
producer.setDeliveryMode(DeliveryMode.PERSISTENT);
String xml = ""+msgCtx.getEnvelope().getBody().toStringWithConsume();
if(topic){
JSONObject obj=XML.toJSONObject(xml);
JSONObject ar=obj.getJSONObject("soapenv:Body");
ar.remove("xmlns:soapenv");
xml=ar.toString();
}
TextMessage message = session.createTextMessage(xml);
producer.send(message);
} catch (Exception e) {
log.info("LogLocation = "+getClass().getName()+",Error in storing message in JMS stacktrace is :"+e.toString()+"message is :"+e.getMessage());
e.printStackTrace();
((Axis2MessageContext) msgCtx).setProperty(NhttpConstants.HTTP_SC, 500);
handleException("Error while storing in the message store", msgCtx);
}
finally {
try {
session.close();
if (connection!=null){
log.info("LogLocation = "+getClass().getName()+",JMS connection closing :"+connection);
connection.close();
}
} catch (JMSException e) {
log.info("LogLocation = "+getClass().getName()+",Error in closing JMS connection stacktrace is :"+e.toString());
e.printStackTrace();
}
}
return true;
}
#Override
public void destroy() {
// TODO Auto-generated method stub
}
#Override
public void init(SynapseEnvironment arg0) {
// TODO Auto-generated method stub
}
}
when i call this service to send a message in queue below logs get generated.
[2017-07-29 11:18:35,962] INFO - JMSStoreMediator LogLocation = in.esb.custommediators.JMSStoreMediator,JMS connection created :ActiveMQConnection {id=ID:my-desktop-36442-1501307315570-3:1,clientId=ID:my-desktop-36442-1501307315570-2:1,started=true}
As of now every thing is working good , But when two users try to submit message at the same tire some strange thing happen as shown below
[2017-07-29 11:43:11,948] INFO - JMSStoreMediator LogLocation = in.my.esb.custommediators.JMSStoreMediator,JMS connection created :ActiveMQConnection {id=ID:my-desktop-36442-1501307315570-11:1,clientId=ID:my-desktop-36442-1501307315570-10:1,started=false}
[2017-07-29 11:43:11,963] INFO - JMSStoreMediator LogLocation = in.my.esb.custommediators.JMSStoreMediator,JMS connection created :ActiveMQConnection {id=ID:my-desktop-36442-1501307315570-11:1,clientId=ID:my-desktop-36442-1501307315570-10:1,started=true}
[2017-07-29 11:43:12,068] INFO - JMSStoreMediator LogLocation = in.my.esb.custommediators.JMSStoreMediator,Error in closing JMS connection stacktrace is :org.apache.activemq.ConnectionClosedException: The connection is already closed
Active MQ is creating two connections but using one connection for both the calls and that one connection is getting closed in one of the service call and throwing already closed error in the other service call and the other connection is waiting forever in the connection list of active mq with active status true as shown in the below image and this is also seen in the ESB thread list.
This kind of connections pileup and causing hangs ESB server. Even if i reset this connections from Active MQ ESB threads carry this connection info and only after a restart of ESB the problem get fixed.
Have you read article Extending the Functionality of WSO2 Enterprise Service Bus - Part 1 ?
Important part is Threading Safety. It states, each mediator, including custom, is shared between incoming messages. I recommend to move class variables
Connection connection;
Session session;
to method public boolean mediate(MessageContext msgCtx) since local variables are thread safe
public class JMSStoreMediator extends AbstractMediator implements
ManagedLifecycle {
public boolean mediate(MessageContext msgCtx) {
Connection connection;
Session session;
....
....
rest the same
Related
I have this method that throws me an exception when the data queue doesn't exists, but is not. Do you have other way to solve this?
public void checkDataQueue(String dataQueue) throws JMSException {
Connection connection = null;
Session session = null;
connection = jmsTemplate.getConnectionFactory().createConnection();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue(dataQueue);
QueueBrowser browser = session.createBrowser(queue);
}
ActiveMQ 5.x creates Queues on demand by default so you may have changed the default configuration to disallow this in which case the error should be expected to happen if you are hitting a non-existent queue and you should check for and handle that. If you need to be sure then the broker provides a JMX interface to query for information on broker statistics etc. There are also other ways of monitoring such as using Rest style calls over the Jolokia management interface.
thank you Tim, I solved it with these method.
public boolean existDataQueue(String dataQueue) throws JMSException {
boolean response = false;
ActiveMQConnectionFactory activeMQConnectionFactory =
new ActiveMQConnectionFactory();
activeMQConnectionFactory.setBrokerURL(brokerUrl);
ActiveMQConnection connection = (ActiveMQConnection)activeMQConnectionFactory.createConnection();
connection.start();
DestinationSource ds = connection.getDestinationSource();
Set<ActiveMQQueue> queues = ds.getQueues();
for (ActiveMQQueue activeMQQueue : queues) {
try {
if(activeMQQueue.getQueueName().equalsIgnoreCase(dataQueue)) {
response = true;
}
} catch (JMSException e) {
e.printStackTrace();
}
}
connection.close();
return response;
}
My application is using ActiveMQ embedded with TomEE Plus-8.0.0-M1.
The behavior of running out of pooled connections is the same regardless of using an explicitly defined javax.jms.ConnectionFactory using resources.xml or letting TomEE generate one on the fly.
Different settings for connection timeout don't seem to have an effect:
connectionMaxIdleTime = 3 seconds
connectionMaxIdleTime = 15 minutes
If ExampleSessionBean.sendMessage() is called 11 times sequentially, Abandoned Connection warnings are logged on messages 1-10. On message #11, the JMSRuntimeException: No Managed Connections Available, is thrown.
Then if I wait a few minutes, the JMSContext producer is able to send again.
My first thought was that the underlying connection in JMSContext needed to be closed with
jmsContext.close();
but from Interface JMSContext
"This method must not be used if the JMSContext is container-managed (injected)."
What programmatic or configuration changes are needed here?
Thank you,
Ted S
resources.xml
<?xml version="1.0" encoding="UTF-8"?>
<resources>
<Resource id="jmsConnectionFactory" type="javax.jms.ConnectionFactory">
connectionMaxIdleTime = 3 seconds
connectionMaxWaitTime = 5 seconds
poolMaxSize = 10
poolMinSize = 0
resourceAdapter = Default JMS Resource Adapter
transactionSupport = xa
</Resource>
</resources>
ExampleSessionBean.java
#Named
#LocalBean
#Stateless
public class ExampleSessionBean
{
#Resource(name = "jms/localNotificationQueue")
private Queue _localNotificationQueue;
#Inject
#JMSConnectionFactory("jmsConnectionFactory")
private JMSContext _jmsContext;
public void sendMessage(String message)
{
try
{
TextMessage textMessage =
_jmsContext.createTextMessage(message);
_jmsContext.createProducer().
setDeliveryMode(DeliveryMode.PERSISTENT).
send(_localNotificationQueue, textMessage);
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
Warnings - Messages 1-10: (Pardon the formatting. SE formatter won't format this text block for some reason)
Begin Warning Text
WARNING: Transaction complete,
but connection still has handles associated: ManagedConnectionInfo:
... org.apache.openejb.resource.activemq.jms2.cdi.JMS2CDIExtension$InternalJMSContext.createProducer(JMS2CDIExtension.java:370),
com.myorg.jms.ExampleSessionBean.sendMessage(ExampleSessionBean.java:46),
...
End Warning Text
Message #11 Exception
javax.jms.JMSRuntimeException: No ManagedConnections available within configured blocking timeout ( 5000 [ms] ) for pool org.apache.geronimo.connector.outbound.SinglePoolConnectionInterceptor#12aff7fa
at org.apache.openejb.resource.activemq.jms2.JMS2.toRuntimeException(JMS2.java:83)
at org.apache.openejb.resource.activemq.jms2.JMSContextImpl.connection(JMSContextImpl.java:85)
at org.apache.openejb.resource.activemq.jms2.JMSContextImpl.session(JMSContextImpl.java:102)
at org.apache.openejb.resource.activemq.jms2.JMSContextImpl.getInnerProducer(JMSContextImpl.java:124)
at org.apache.openejb.resource.activemq.jms2.JMSContextImpl.createProducer(JMSContextImpl.java:302)
at org.apache.openejb.resource.activemq.jms2.cdi.JMS2CDIExtension$InternalJMSContext.createProducer(JMS2CDIExtension.java:370)
at com.myorg.jms.ExampleSessionBean.sendMessage(ExampleSessionBean.java:46)
...
UPDATE: This behavior has been duplicated and added to the TomEE issues tracker.
I gave up on JMSContext & went back to manually managing the connection. All functions as expected even when loading up with 100s of messages.
ExampleSessionBean.java
#Named
#LocalBean
#Stateless
public class ExampleSessionBean
{
#Resource(name = "jms/localNotificationQueue")
private Queue _localNotificationQueue;
#Resource(mappedName = "jmsConnectionFactory")
private ConnectionFactory _connectionFactory;
public void sendMessage(String message)
{
Connection connection = null;
Session session = null;
MessageProducer messageProducer = null;
try
{
connection =
_connectionFactory.createConnection();
connection.start();
session =
connection.createSession();
messageProducer =
session.createProducer(null);
messageProducer.setDeliveryMode(DeliveryMode.PERSISTENT);
TextMessage textMessage = session.createTextMessage(message);
messageProducer.send(_localNotificationQueue,
textMessage);
}
catch (Exception e)
{
// TODO Handle exception
}
finally
{
if (connection != null)
{
try
{
connection.close();
}
catch (JMSException e) {}
}
}
}
}
In this code, I am using setJMSExpiration(1000) for expire message of one second in queue from publisher side. But From Consumer Side, It is returning properly message after 1 second instead of null.
public class RegistrationPublisher extends Thread{
public void run() {
publisherQueue("Registration.Main.*");
}
public void publisherQueue(String server){
try {
String url="tcp://192.168.20.49:61616";
// Create a ConnectionFactory
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory(url);
Connection connection = connectionFactory.createConnection();
connection.start();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Destination destination = session.createQueue(server);
MessageProducer producer = session.createProducer(destination);
producer.setDeliveryMode(DeliveryMode.PERSISTENT);
String text = "Test";
TextMessage message = session.createTextMessage(text);
message.setJMSExpiration(1000);// For Expire message in one second
producer.send(message);
producer.close();
session.close();
connection.close();
}
catch (Exception e) {
e.printStackTrace();
}
}
public static void main(String args[]) throws IOException{
RegistrationPublisher registrationPublisher=new RegistrationPublisher();
registrationPublisher.start();
}
}
You do this by configuring the JMS MessageProducer to do it for you via the send method that accepts a TTL or by calling setTimeToLive on the producer which adds the same TTL to all sent messages. The JMS APIs for the message version are clear that calling the setters on the message have no effect.
void setJMSExpiration(long expiration) throws JMSException
Sets the message's expiration value.
This method is for use by JMS providers only to set this field when a message is sent. This message cannot be used by clients to configure the expiration time of the message. This method is public to allow a JMS provider to set this field when sending a message whose implementation is not its own.
I first also thought that is was possible to set expiration directly on the message in the post-processor, but as Tim Bish said above, this is not the intended way to do it, and the value will get reset to 0 afterward. I couldn't access to the producer directly neither to set a time to live, because this object was in library org.springframework.jms (I was following this documentation).
One thing I could do was to set to time to live on the jmsTemplate:
import org.springframework.jms.core.JmsTemplate;
#Service
public class MyJmsServiceImpl implements MyJmsService {
#Inject
private JmsTemplate jmsTemplate;
private void convertAndSendToResponseQueue(String targetQueueName, String correlationid, Object message) {
// Set time to live
jmsTemplate.setExplicitQosEnabled(true);
jmsTemplate.setTimeToLive(5000);
jmsTemplate.convertAndSend(targetQueueName, message, new JmsResponsePostProcessor(correlationid));
}
}
I tried to create a log4j filter that sends all error logs from my tomcat server in a JMS QUEUE and from there to do whatever I want with them.
First I test the functionality because is first time I am working with ActiveMQ, first time I am trying to make a log4j filter:
I added a JMSAppender to log4j2.xml as specified here
<Appenders>
<JMS name="jmsQueue" destinationBindingName="errorLogQueue"
factoryName="org.apache.activemq.jndi.ActiveMQInitialContextFactory"
factoryBindingName="ConnectionFactory"
providerURL="tcp://localhost:61616"/>
</Appenders>
<Loggers>
<Root level="error">
<AppenderRef ref="jmsQueue"/>
</Root>
</Loggers>
and I created a class to test the functionality:
public class Log4JjmsAppenderWithActiveMQ implements MessageListener, Runnable{
public static Logger logger2 = LogManager.getLogger(Log4JjmsAppenderWithActiveMQ.class);
public static void main(String[] args) {
(new Thread(new Log4JjmsAppenderWithActiveMQ())).start();
}
#Override
public void run() {
ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory("tcp://localhost:61616");
factory.setTrustAllPackages(true);
Connection connection = null;
Session session = null;
Destination destination = null;
MessageConsumer consumer = null;
try {
connection = factory.createConnection();
connection.start();
session = connection.createSession(false, session.AUTO_ACKNOWLEDGE);
destination = session.createQueue("errorLogQueue");
consumer = session.createConsumer(destination);
consumer.setMessageListener(this);
logger2.error("error 1");
logger2.error("error 2");
logger2.info("info 1");
logger2.debug("debug 1");
} catch (JMSException | InterruptedException e) {
e.printStackTrace();
} finally {
try {
consumer.close();
session.close();
connection.close();
} catch (JMSException e) {
e.printStackTrace();
}
}
}
#Override
public void onMessage(Message message) {
Log4jLogEvent event = null;
try {
event = (Log4jLogEvent)((ActiveMQObjectMessage)message).getObject();
} catch (JMSException e) {
e.printStackTrace();
}
LocalDate date = Instant.ofEpochMilli(event.getTimeMillis()).atZone(ZoneId.systemDefault()).toLocalDate();
System.out.println("Received log [" + event.getLevel() + "]: "+ event.getMessage()+" at time: "+date);
}
}
Running the app I get what I expected. Only errors are logged:
Received log [ERROR]: error 1
Received log [ERROR]: error 2
and also the ActiveMQ says that from 4 logged infos the two with level error were enqueued and dequeued error message.
Well, is first time I am working with ActiveMQ and also I don't know much about configuring servers.
So, my test worked but now I want to run this app when the tomcat starts and to send in the queue the errors that are on the server.
I think I have to change the providerURL value. If so, as host should I put my server host and for port should I use the port used for my server?
My #WebListener class implements ServletContextListener and in it's contextInitialized() method I configure the JMS server by calling the method bellow:
private void configActiveMQServer() {
ActiveMQConnectionFactory factory = new
ActiveMQConnectionFactory("tcp://localhost:61616");
factory.setTrustAllPackages(true);
Connection connection = null;
Session session = null;
Destination destination = null;
try {
connection = factory.createConnection();
connection.start();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
destination = session.createQueue("errorLogQueue");
} catch (JMSException e) {
e.printStackTrace();
}
}
well, logging on the server the errors doesn't get into the queue.
Maybe the jmsAppender is not the one I am looking for...
So, I am trying to create a custom appender by extending AppenderSkeleton following examples lke: this and this documentation - search for "Appenders" in the documentation to see.
For now, my custom appender looks like:
package com.h4p.server;
import java.io.Serializable;
import org.apache.log4j.AppenderSkeleton;
import org.apache.log4j.Level;
import org.apache.log4j.spi.LoggingEvent;
import org.apache.logging.log4j.core.Appender;
import org.apache.logging.log4j.core.Filter;
import org.apache.logging.log4j.core.Layout;
import org.apache.logging.log4j.core.config.plugins.Plugin;
import org.apache.logging.log4j.core.config.plugins.PluginAliases;
import org.apache.logging.log4j.core.config.plugins.PluginAttribute;
import org.apache.logging.log4j.core.config.plugins.PluginElement;
import org.apache.logging.log4j.core.config.plugins.PluginFactory;
import org.apache.logging.log4j.core.layout.PatternLayout;
#Plugin(name = Log4JErrorAppender.PLUGIN_NAME, category = "Core", elementType = Appender.ELEMENT_TYPE, printObject = true)
#PluginAliases(value="Log4JErrorAppender")
public class Log4JErrorAppender extends AppenderSkeleton{
public static final String PLUGIN_NAME = "Log4JErrorAppender";
#Override
public void close() {
System.out.println("xxx");
}
#Override
public boolean requiresLayout() {
return false;
}
#Override
protected void append(LoggingEvent event) {
if(event.getLevel() == Level.ERROR) {
}
}
#PluginFactory
public static Log4JErrorAppender createAppender(#PluginAttribute("name") String name,
#PluginAttribute("ignoreExceptions") boolean ignoreExceptions,
#PluginElement("Layout") Layout<? extends Serializable> layout,
#PluginElement("Filters") Filter filter) {
if (name == null) {
System.out.println("No name provided for Log4JErrorAppender");
return null;
}
if (layout == null) {
layout = PatternLayout.createDefaultLayout();
}
return new Log4JErrorAppender();
}
}
and in log4j2.xml I declared:
<Log4JErrorAppender name="errorQueue"/>
but I have this error:
localhost-startStop-1 ERROR appenders Appenders has no parameter that matches element Log4JErrorAppender
Could you please give me a good guide to create a custom appender in order to satisfy my need?
The main question is how to get the errors from my server and send (append) them to a queue from where I can consume the error messages from a client in another project? What log4j appender should I use? Give me some examples.
Thanks!
Everything you did is fine. To break it down, these are the components of your "system":
Log producer - Tomcat that runs an application that produces logs and publishes them to ActiveMQ. Note here, JMSAppender is exactly what you need. You can see an example in the official ActiveMQ documentation: How to use JMS Appender
ActiveMQ instance - you need to set up a standalone ActiveMQ server.
I think I have to change the providerURL value. If so, as host should I put my server host and for port should I use the port used for my server?
You need to provide the host and port as parameters to the JMS Appender (exactly as you did in the log4j xml config).
Log consumer - this can the same application as in (1) or a different one. The correct way to consume messages is by implementing a MessageListener and handling incoming messages in onMessage(..)
As it seems, it's not clear for you how to bootstrap the last component (3). What you need to do is to initialize your message listener inside contextInitialized(...). When you do that, you will have to bind it to a specific topic/queue. This needs to be the same as the one you publish logs to in (1). You can see a brief example below:
public class LogMessageListenerExample implements ServletContextListener
{
#Override
public void contextInitialized(ServletContextEvent arg0)
{
ConnectionFactory conn = createConnectionFactory();
Session session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
conn.start();
MessageConsumer consumer = session.createConsumer(sess.createTopic("topic/queue-name-to-listen"));
LogMessageListener listener = new LogMessageListener();
consumer.setMessageListener(listener);
}
private ConnectionFactory createConnectionFactory()
{
// create connection factory with host and port for your ActiveMQ instance
}
}
public class LogMessageListener implements MessageListener
{
public void onMessage(Message message)
{
// handle message
}
}
I am listening to a queue of activemq.
My configuration is like below:
ACTIVEMQ_BROKER_URL =failover:(tcp://LON6QAEQTAPP01:61616?wireFormat.maxInactivityDuration=0)
From the console logs I can see that a reconnect attempt is made to connect to this server.
[2014-08-20 08:57:43,303] INFO 236[ActiveMQ Task-1] -
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1030)
- Successfully connected to tcp://LON6QAEQTAPP01:61616?wireFormat.maxInactivityDuration=0
[2014-08-20 08:57:43,355] INFO 288[ActiveMQ Task-1] -
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1030)
- Successfully connected to tcp://LON6QAEQTAPP01:61616?wireFormat.maxInactivityDuration=0
[2014-08-20 08:57:43,374] INFO 307[ActiveMQ Task-1] -
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1030)
- Successfully connected to tcp://LON6QAEQTAPP01:61616?wireFormat.maxInactivityDuration=0
Still I am not able to consume the message. I am using the following code to try and handle it from code side but still not able to get the connection persistent.
try
{
connection.start();
while (true)
{
try
{
if (connection.isClosed() || connection.isTransportFailed() || session.isClosed() || !connection.getTransport().isConnected())
{
log.info("Connection was reset, re-connecting..");
connection.cleanup();
init();
connection.start();
}
}
catch (Throwable t)
{
init();
log.error("Connection was reset, re-connecting in exception block", t);
}
Thread.sleep(30000L);
}
private void init() throws Exception
{
init(brokerURL, queueName, userName, password, manual);
}
public void init(String brokerURL, String queueName, String userName, String password, boolean manual) throws Exception
{
this.manual = manual;
this.brokerURL = brokerURL;
this.queueName = queueName;
this.userName = userName;
this.password = password;
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(userName, password, brokerURL);
connection = (ActiveMQConnection) connectionFactory.createConnection();
connection.addTransportListener(new TransportListener()
{
#Override
public void transportResumed()
{
log.info("Transport Resumed ");
}
#Override
public void transportInterupted()
{
log.info("Transport Interupted ");
}
#Override
public void onException(IOException arg0)
{
arg0.printStackTrace();
log.error("Transport Exception: " + arg0.getMessage());
}
#Override
public void onCommand(Object arg0)
{
// TODO Auto-generated method stub
}
});
connection.setExceptionListener(this);
connection.setCloseTimeout(10);
session = (ActiveMQSession) connection.createSession(false, Session.AUTO_ACKNOWLEDGE);// CLIENT_ACKNOWLEDGE
Destination destination = session.createQueue(queueName);
MessageConsumer consumer = session.createConsumer(destination);
consumer.setMessageListener(this);
}
Can someone help me as to how can I get this connection to be active alive all the time. It generally times out after inactivity of 40 mins or so.
Trying to force reconnect an open connection the way you are is definitely not going to do you and good. You need to tear down the connection and create a new one if you want to handle connection interruption yourself. What you really should do is figure out what is closing your connection on you, probably a firewall closing down the inactive connection or a load balancer. Look at your environment and see what else is in the mix that could be closing the connection on you.
The broker can, if configured to do so, rebalance clients in a cluster. Are you using multiple clients in a clustered environment etc?