i want to do a throttling to a consumer of some queue in the activeMQ, in hornetq (of jboss, this is do it with annotations on the definition of the mdb Consumer). I can't find any similar in the documentation of activemq, the closest that i find was this
consumer.recvDelay 0 ms Pause consumer for recvDelay milliseconds with each message (allows consumer throttling).
from: http://activemq.apache.org/activemq-performance-module-users-manual.html
But there i can't find how i can do it in java.
Thanks in advance,
Regards.
EDIT: Here is the ActiveMQManager code and the consumer code:
public class ActiveMQManager {
private static ActiveMQConnectionFactory CONNECTION_FACTORY;
public static Connection CONNECTION;
public static Session SESSION;
public static Destination TEST_QUEUE;
public static void start() {
try {
CONNECTION_FACTORY = new ActiveMQConnectionFactory("vm://localhost");
CONNECTION = CONNECTION_FACTORY.createConnection();
CONNECTION.start();
SESSION = CONNECTION.createSession(false,
Session.CLIENT_ACKNOWLEDGE);
TestClient testClient = new TestClient();
TEST_QUEUE = SESSION.createQueue("TEST.QUEUE");
MessageConsumer testConsumer = SESSION.createConsumer(TEST_QUEUE);
test.setMessageListener(testClient);
} catch (Exception e) {
}
}
public static void stop() {
try {
// Clean up
SESSION.close();
CONNECTION.close();
} catch (JMSException e) {
log.error(e);
}
}
}
The consumer code is very simple (for this example):
public class TestConsumer implements MessageListener {
#Override
public void onMessage(Message message) {
//Do something with the message
}
}
this depends on the consumer technology being used...but here are a few options
you can manually introduce a delay in your consumer code (not an exact science but this will limit the throughput)
you can also control the number of threads that your consumer uses by setting maxConcurrentConsumers property of you JMS connection...that said, this won't throttle message throughput, just limit the level of concurrency being used by your consumer
better yet, you can set the exact number of messages to consume per time period using a throttler EIP implementation
for example, this is trivial using the Camel Throttler
from("activemq:queueA").throttle(10).to("activemq:queueB")
With ActiveMQ you can set the consumer prefetch limit: http://activemq.apache.org/what-is-the-prefetch-limit-for.html
To configure it, you can use the connection URL (most of the configurations can be done using the URL) or the Java API.
For more interesting parameters: http://activemq.apache.org/connection-configuration-uri.html
Take into account that Camel Throttler keeps the exchanges in-memory while they are blocked by the Throttler. So, if you have 100 queue consumers and the Server (exposing the SOAP service) is slow, you may have in-memory up to 100 exchanges!
Posted this question for this: Throttle consumption rate of all JMS consumers listening on an ActiveMQ queue
Related
I'm using hazel cast IMGD for my app. I have used queues for internal communication. I added an item listener to queue and it works great. Whenever a queue gets a message, listener wakes up and needed processing is done.
Problem is its single threaded. Sometimes, a message takes 30 seconds to process and messages in queue just have to wait until previous message is done processing. I'm told to use Java executor service to have a pool of threads and add an item listener to every thread so that multiple messages can be processed at same time.
Is there any better way to do it ? may be configure some kind of MDB or make the processing asynchronous so that my listener can process the messages faster
#PostConstruct
public void init() {
logger.info(LogFormatter.format(BG_GUID, "Starting up GridMapper Queue reader"));
HazelcastInstance hazelcastInstance = dc.getInstance();
queue = hazelcastInstance.getQueue(FactoryConstants.QUEUE_GRIDMAPPER);
queue.addItemListener(new Listener(), true);
}
class Listener implements ItemListener<QueueMessage> {
#Override
public void itemAdded(ItemEvent<QueueMessage> item) {
try {
QueueMessage message = queue.take();
processor.process(message.getJobId());
} catch (Exception ex) {
logger.error(LogFormatter.format(BG_GUID, ex));
}
}
#Override
public void itemRemoved(ItemEvent<QueueMessage> item) {
logger.info("Item removed: " + item.getItem().getJobId());
}
}
Hazelcast IQueue does not support asynchronous interface. Anyway, asynchronous access would not be faster. MDB requires JMS, which is pure overhead.
What you really need is multithreaded executor. You can use default executor:
private final ExecutorService execService = ForkJoinPool.commonPool();
I am trying to connect to solace queues and simply read messages from them. However, I am able to read the messages but the messages are not getting removed from the queue.
Below is the code which I tried:
public void clearMessages() throws Exception {
// Programmatically create the connection factory using default settings
// Create connection to the Solace router
SolXAConnectionFactoryImpl connectionFactory = returnConnFactory();
XAConnection connection = connectionFactory.createXAConnection();
XASession session = connection.createXASession();
Queue queue = session.createQueue(QUEUE_NAME);
connection.start();
MessageConsumer messageConsumer = session.createConsumer(queue);
//session.b
messageConsumer.setMessageListener(new MessageListener() {
#Override
public void onMessage(Message message) {
if(message instanceof SolTextMessage) {
SolTextMessage solTextMessage =(SolTextMessage)message;
try {
System.out.println("Message cleared is : "+solTextMessage.getText());
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
else {
System.out.println("Message Content: %s"+ SolJmsUtility.dumpMessage(message));
}
latch.countDown();
}
});
latch.await(120,TimeUnit.SECONDS);
connection.stop();
messageConsumer.close();
session.close();
connection.close();
}
Here latch is the object of CountDownLatch which is initialized as:
CountDownLatch latch = new CountDownLatch(2);
You need to commit the XA transaction in order for messages to be consumed.
In JMS, the function call is XAResource.commit(xid, true)
Also, is there a reason to use the CountDownLatch?
If you would like to consume messages synchronously, you can choose not to set a message listener and call MessageConsumer.receive()
Solace does provide a basic sample showing how to make use of XA transactions.
Refer to XATransactions.java in the samples directory of the API.
Note that the sample code is manually managing the XA transaction by calling the relevant XAResource methods such as XAResource.commit().
XA transactions are usually used within Java EE application servers that contain a transaction manager to manage the lifecycle of XA transactions.
I have a Spring JMS Application that is using ActiveMQ version 5.10. I am performing a simple test to concurrency. I am using Spring Boot, current version and annotations to configure JMSListener and message producers.
The message producer just throws messsages on a queue as fast as it can. The message listener is pulling messages off the queue, but sleeping for 1 second after getting the message -- simulating some work that the message listener would need to do after getting a message.
I have the JMSListener set to 100-1000 concurrent threads. If I start the message producer and consumer at the same time (both run in their own JVM) the consumer never gets above the minimum configured threads, even though the max range is set 1000.
If I let the producer start first and place a few thousand messages on the queue, then start 1 or more instances of the consumer, it will raise the threads steadily, starting at 100 then 20 or so threads each second until it gets to a state where there is about 20-30 messages in the queue that are in-flight. It never catches the producer -- there is always some messages in queue even though the consumer is no where near its maxConcurrency count.
Why doesn't the message consumer burst into a bunch of additional threads to empty the queue instead of letting the queue have the 20-30 messages in it? Isn't there a way for the consumer continue to add threads faster in order to catch up with the messages in queue?
Here are the relevant parts of the code.
Message Producer
#Component
public class ClientServiceImpl implements ClientService {
private static final String QUEUE="message.test.queue";
#Autowired
private JmsTemplate jmsTemplate;
#Override
public void submitMessage(ImportantMessage importantMessage) {
System.out.println("*** Sending " + importantMessage);
jmsTemplate.convertAndSend(QUEUE, importantMessage);
}
}
Message Consumer
#SpringBootApplication
#EnableJms
public class AmqConsumerApplication {
public static void main(String[] args) {
SpringApplication.run(AmqConsumerApplication.class, args);
}
#Value("${JMSHost}")
private String JMS_BROKER_URL;
#Autowired
static Command command;
#Bean
public ConnectionFactory connectionFactory() {
ConnectionFactory factory= new ActiveMQConnectionFactory(JMS_BROKER_URL);
((ActiveMQConnectionFactory)factory).setTrustAllPackages(true);
((ActiveMQConnectionFactory)factory).setOptimizeAcknowledge(true);
((ActiveMQConnectionFactory)factory).setAlwaysSessionAsync(false);
return factory;
}
}
With the listener configured as such...
#Component
public class TransformationListener {
private static final String QUEUE="message.test.queue?consumer.prefetchSize=10";
#JmsListener(destination=QUEUE, concurrency = "100-1000")
public void handleRequest(ImportantMessage importantMessage) {
System.out.println("*** Recieved message: " + importantMessage + " on thread" + Thread.currentThread().getId());
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
Are you still facing this behavior ?
Did you read this advice "Pooled Consumers and prefetch" on http://activemq.apache.org/what-is-the-prefetch-limit-for.html
Did you tried with prefetchSize=0 or 1 ? I think 1 can resolve your problem.
If prefetchSize is > 1 maybe you need to decrease the AbortSlowAckConsumerStrategy to lower than default 30s.
To have more than 100 threads consuming messages in your case you need more than 1000 messages not consumed and not prefetched in the queue because the prefetchSize is to 10.
I have two components, the first one sends AMQ messages to second one using Apache Camel(Request/Response Pattern), but some of messages don't get consumed by second component(remains pending. after some minutes, that message goes to dlQueue).
Why?
this is some of my code:
public void boot() {
try {
main = new Main();
main.enableHangupSupport();
PooledConnectionFactory pooledConnectionFactoryBean = new PooledConnectionFactory();
pooledConnectionFactoryBean.setMaxConnections(8);
pooledConnectionFactoryBean.setMaximumActiveSessionPerConnection(500);
ActiveMQComponent comp1 = activeMQComponent("failover:(tcp://localhost:61616)?maxReconnectDelay=40000&warnAfterReconnectAttempts=1");
comp1.setUsePooledConnection(true);
comp1.setConnectionFactory(pooledConnectionFactoryBean.getConnectionFactory());
main.bind("activemq", comp1);
main.bind("activemqException", activeMQComponent("failover:(tcp://localhost:61616)?maxReconnectDelay=40000&warnAfterReconnectAttempts=1"));
main.addRouteBuilder(new MyRouteBuilder());
main.run();
} catch (Exception e) {
log.error("Exception", e);
}
}
AND the RouteBuilder of second component:
private static class MyRouteBuilder extends RouteBuilder {
#Override
public void configure() {
//...
from("activemq:queue:Q.second")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
//...
}
})
.log("The Reply of (${in.header.Title}) has been sent successfully!");
}
}
If I remember right the default configuration of the ActiveMQ endpoint applies a prefetch limit of 1000 messages. Which can cause the problem you describe in combination with pooled connections.
Using a single connection instead of a connection pool for your ActiveMQ consumer endpoint or setting the prefetch limit to 0 is likely to fix the problem.
Setting the prefetch limit to 0 for all consumers can be achieved by appending jms.prefetchPolicy.all=0 to your connection URL.
Check out the AMQ documentation on prefetch.
I am using Spring's message-driven POJO framework (and DefaultMessageListenerContainer in particular) to listen to several queues and topics.
In the case of one particularly queue, there is a need to slow the rate at which I drain the queue, on the order of one message every five minutes. The actual processing of the messages is a sub-second operation, but I would like the listener to sit idle for some time in between messages.
I have created a bit of a hack, but it is decidedly sub-optimal: What I've done is to set the max concurrency to 1 and add a Thread.sleep(..) after processing each message. I would like to find a way instead to use the DefaultMessageListenerContainer to wait between attempts to receive, rather than causing the handler to do the waiting during the would-be processing of a message.
I had considered if there was a ScheduledExecutor that would help, but I realize that the throttling would need to be done where the tasks are produced. Is there perhaps some method from DefaultMessageListenerContainer that I could override to accomplish what I'm after?
Depending on the provider of the queue, you may be able to set a max rate for consumers that consume it's queues.
For example in hornetQ you set this in the connection factory using consumer-max-rate.
An alternative to modifying the behavior of your consumer would be to make use of Apache Camel to delay the messages on that one specific queue.
http://camel.apache.org/delayer.html describes the functionality of the Camel Delayer pattern. So for example:
<route>
<from uri="jms:YOURQUEUE"/>
<delay>
<constant>1000</constant>
</delay>
<to uri="jms:DELAYEDQUEUE"/>
</route>
Where you would then consume the DELAYEDQUEUE and all messages would be delayed by 1 second.
I'm not sure for 100%, but believe that receiveTimeout is what you want.
<bean id="blahContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
....
<!-- 300000 = 5 * 60 * 1000 == 5 min -->
<property name="receiveTimeout" value="300000"/>
</bean>
receiveTimeout accepts timeout in milliseconds, you can read more about it in javadocs
Here's a solution that extends DefaultMessageListenerContainer to provide the throttling functionality. The advantage of this approach is that Thread.sleep() is not being called within onMessage(). This would hold a Transaction open for longer than necessary if a Transaction is in play (as configured in this example below). The call to Thread.sleep() occurs after the transaction has been committed. A limitation to implementing this throttling feature is that we can only support one consumer thread, hence the name ThrottlingSingleConsumerMessageListenerContainer.
#Configuration
#EnableJms
#EnableTransactionManagement
public class Config
{
private static final long THROTTLE_FIVE_SECONDS = 5_000;
#Bean
public DefaultMessageListenerContainer defaultMessageListenerContainer(
ConnectionFactory connectionFactory,
PlatformTransactionManager transactionManager,
MyJmsListener myJmsListner)
{
DefaultMessageListenerContainer dmlc = new ThrottlingSingleConsumerMessageListenerContainer(THROTTLE_FIVE_SECONDS);
dmlc.setConnectionFactory(connectionFactory);
dmlc.setSessionAcknowledgeMode(Session.SESSION_TRANSACTED);
dmlc.setSessionTransacted(true);
dmlc.setTransactionManager(transactionManager);
dmlc.setDestinationName("QUEUE.IN");
dmlc.setMessageListener(myJmsListner);
return dmlc;
}
}
#Component
public class MyJmsListener implements MessageListener
{
#Override
public void onMessage(Message message)
{
// process the message
}
}
public class ThrottlingSingleConsumerMessageListenerContainer extends DefaultMessageListenerContainer
{
private static final Logger log = LoggerFactory.getLogger(ThrottlingSingleConsumerMessageListenerContainer.class);
private final long delayMillis;
public ThrottlingSingleConsumerMessageListenerContainer(long delayMillis)
{
this.delayMillis = delayMillis;
super.setMaxConcurrentConsumers(1);
}
#Override
protected boolean receiveAndExecute(Object invoker, #Nullable Session session, #Nullable MessageConsumer consumer) throws JMSException
{
boolean messageReceived = super.receiveAndExecute(invoker, session, consumer);
if (messageReceived) {
log.info("Sleeping for {} millis", delayMillis);
try {
Thread.sleep(delayMillis);
} catch (InterruptedException e) {
log.warn("Sleeping thread has been interrupted");
Thread.currentThread().interrupt();
}
}
return messageReceived;
}
#Override
public void setMaxConcurrentConsumers(int maxConcurrentConsumers)
{
super.setMaxConcurrentConsumers(maxConcurrentConsumers);
Assert.isTrue(getMaxConcurrentConsumers() <= 1, "Throttling does not support maxConcurrentConsumers > 1");
}
#Override
public void setConcurrency(String concurrency)
{
super.setConcurrency(concurrency);
Assert.isTrue(getMaxConcurrentConsumers() <= 1, "Throttling does not support maxConcurrentConsumers > 1");
}
}
This has been tested on org.springframework 5.x but should run on earlier versions also.