I have two components, the first one sends AMQ messages to second one using Apache Camel(Request/Response Pattern), but some of messages don't get consumed by second component(remains pending. after some minutes, that message goes to dlQueue).
Why?
this is some of my code:
public void boot() {
try {
main = new Main();
main.enableHangupSupport();
PooledConnectionFactory pooledConnectionFactoryBean = new PooledConnectionFactory();
pooledConnectionFactoryBean.setMaxConnections(8);
pooledConnectionFactoryBean.setMaximumActiveSessionPerConnection(500);
ActiveMQComponent comp1 = activeMQComponent("failover:(tcp://localhost:61616)?maxReconnectDelay=40000&warnAfterReconnectAttempts=1");
comp1.setUsePooledConnection(true);
comp1.setConnectionFactory(pooledConnectionFactoryBean.getConnectionFactory());
main.bind("activemq", comp1);
main.bind("activemqException", activeMQComponent("failover:(tcp://localhost:61616)?maxReconnectDelay=40000&warnAfterReconnectAttempts=1"));
main.addRouteBuilder(new MyRouteBuilder());
main.run();
} catch (Exception e) {
log.error("Exception", e);
}
}
AND the RouteBuilder of second component:
private static class MyRouteBuilder extends RouteBuilder {
#Override
public void configure() {
//...
from("activemq:queue:Q.second")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
//...
}
})
.log("The Reply of (${in.header.Title}) has been sent successfully!");
}
}
If I remember right the default configuration of the ActiveMQ endpoint applies a prefetch limit of 1000 messages. Which can cause the problem you describe in combination with pooled connections.
Using a single connection instead of a connection pool for your ActiveMQ consumer endpoint or setting the prefetch limit to 0 is likely to fix the problem.
Setting the prefetch limit to 0 for all consumers can be achieved by appending jms.prefetchPolicy.all=0 to your connection URL.
Check out the AMQ documentation on prefetch.
Related
I have a scenario where I am establishing TCP connection using netty NIO, suppose server went down than how can I automatically connect to server when it comes up again ?
Or Is there any way to attach availability listener on server ?
You can have a DisconnectionHandler, as the first thing on your client pipeline, that reacts on channelInactive by immediately trying to reconnect or scheduling a reconnection task.
For example,
public class DisconnectionHandler extends ChannelInboundHandlerAdapter {
#Override
public void channelInactive(final ChannelHandlerContext ctx) throws Exception {
Channel channel = ctx.channel();
/* If shutdown is on going, ignore */
if (channel.eventLoop().isShuttingDown()) return;
ReconnectionTask reconnect = new ReconnectionTask(channel);
reconnect.run();
}
}
The ReconnectionTask would be something like this:
public class ReconnectionTask implements Runnable, ChannelFutureListener {
Channel previous;
public ReconnectionTask(Channel c) {
this.previous = c;
}
#Override
public void run() {
Bootstrap b = createBootstrap();
b.remoteAddress(previous.remoteAddress())
.connect()
.addListener(this);
}
#Override
public void operationComplete(ChannelFuture future) throws Exception {
if (!future.isSuccess()) {
// Will try to connect again in 100 ms.
// Here you should probably use exponential backoff or some sort of randomization to define the retry period.
previous.eventLoop()
.schedule(this, 100, MILLISECONDS);
return;
}
// Do something else when success if needed.
}
}
Check here for an example of Exponential Backoff library.
I have a Spring JMS Application that is using ActiveMQ version 5.10. I am performing a simple test to concurrency. I am using Spring Boot, current version and annotations to configure JMSListener and message producers.
The message producer just throws messsages on a queue as fast as it can. The message listener is pulling messages off the queue, but sleeping for 1 second after getting the message -- simulating some work that the message listener would need to do after getting a message.
I have the JMSListener set to 100-1000 concurrent threads. If I start the message producer and consumer at the same time (both run in their own JVM) the consumer never gets above the minimum configured threads, even though the max range is set 1000.
If I let the producer start first and place a few thousand messages on the queue, then start 1 or more instances of the consumer, it will raise the threads steadily, starting at 100 then 20 or so threads each second until it gets to a state where there is about 20-30 messages in the queue that are in-flight. It never catches the producer -- there is always some messages in queue even though the consumer is no where near its maxConcurrency count.
Why doesn't the message consumer burst into a bunch of additional threads to empty the queue instead of letting the queue have the 20-30 messages in it? Isn't there a way for the consumer continue to add threads faster in order to catch up with the messages in queue?
Here are the relevant parts of the code.
Message Producer
#Component
public class ClientServiceImpl implements ClientService {
private static final String QUEUE="message.test.queue";
#Autowired
private JmsTemplate jmsTemplate;
#Override
public void submitMessage(ImportantMessage importantMessage) {
System.out.println("*** Sending " + importantMessage);
jmsTemplate.convertAndSend(QUEUE, importantMessage);
}
}
Message Consumer
#SpringBootApplication
#EnableJms
public class AmqConsumerApplication {
public static void main(String[] args) {
SpringApplication.run(AmqConsumerApplication.class, args);
}
#Value("${JMSHost}")
private String JMS_BROKER_URL;
#Autowired
static Command command;
#Bean
public ConnectionFactory connectionFactory() {
ConnectionFactory factory= new ActiveMQConnectionFactory(JMS_BROKER_URL);
((ActiveMQConnectionFactory)factory).setTrustAllPackages(true);
((ActiveMQConnectionFactory)factory).setOptimizeAcknowledge(true);
((ActiveMQConnectionFactory)factory).setAlwaysSessionAsync(false);
return factory;
}
}
With the listener configured as such...
#Component
public class TransformationListener {
private static final String QUEUE="message.test.queue?consumer.prefetchSize=10";
#JmsListener(destination=QUEUE, concurrency = "100-1000")
public void handleRequest(ImportantMessage importantMessage) {
System.out.println("*** Recieved message: " + importantMessage + " on thread" + Thread.currentThread().getId());
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
Are you still facing this behavior ?
Did you read this advice "Pooled Consumers and prefetch" on http://activemq.apache.org/what-is-the-prefetch-limit-for.html
Did you tried with prefetchSize=0 or 1 ? I think 1 can resolve your problem.
If prefetchSize is > 1 maybe you need to decrease the AbortSlowAckConsumerStrategy to lower than default 30s.
To have more than 100 threads consuming messages in your case you need more than 1000 messages not consumed and not prefetched in the queue because the prefetchSize is to 10.
I implementing websockets using Vert.x 3.
The scenario is simple: opening socket from client doing some 'blocking' work at the vertex verticle worker and when finish response with the answer to the client(via the open socket)
Please tell me if I am doing it right:
Created VertxWebsocketServerVerticle. as soon as the websocket is opening and request coming from the client I am using eventBus and passing the message to
EventBusReceiverVerticle. there I am doing blocking operation.
how I am actually sending back the response back to VertxWebsocketServerVerticle and sending it back to the client?
code:
Main class:
public static void main(String[] args) throws InterruptedException {
Vertx vertx = Vertx.vertx();
vertx.deployVerticle(new EventBusReceiverVerticle("R1"),new DeploymentOptions().setWorker(true));
vertx.deployVerticle(new VertxWebsocketServerVerticle());
}
VertxWebsocketServerVerticle:
public class VertxWebsocketServerVerticle extends AbstractVerticle {
public void start() {
vertx.createHttpServer().websocketHandler(webSocketHandler -> {
System.out.println("Connected!");
Buffer buff = Buffer.buffer().appendInt(12).appendString("foo");
webSocketHandler.writeFinalBinaryFrame(buff);
webSocketHandler.handler(buffer -> {
String inputString = buffer.getString(0, buffer.length());
System.out.println("inputString=" + inputString);
vertx.executeBlocking(future -> {
vertx.eventBus().send("anAddress", inputString, event -> System.out.printf("got back from reply"));
future.complete();
}, res -> {
if (res.succeeded()) {
webSocketHandler.writeFinalTextFrame("output=" + inputString + "_result");
}
});
});
}).listen(8080);
}
#Override
public void stop() throws Exception {
super.stop();
}
}
EventBusReceiverVerticle :
public class EventBusReceiverVerticle extends AbstractVerticle {
private String name = null;
public EventBusReceiverVerticle(String name) {
this.name = name;
}
public void start(Future<Void> startFuture) {
vertx.eventBus().consumer("anAddress", message -> {
System.out.println(this.name +
" received message: " +
message.body());
try {
//doing some looong work..
Thread.sleep(10000);
System.out.printf("finished waiting\n");
startFuture.complete();
} catch (InterruptedException e) {
e.printStackTrace();
}
});
}
}
I always get:
WARNING: Message reply handler timed out as no reply was received - it will be removed
github project at: https://github.com/IdanFridman/VertxAndWebSockets
thank you,
ray.
Since you are blocking your websocket handler until it receives a reply for the sent message to the EventBus, which will not, in fact, be received until the set up delay of 10s laps, you certainly will get warning since the reply handler of the event bus will timeout -> Message sent but no response received before the timeout delay.
Actually I don't know if you are just experimenting the Vert.x toolkit or you are trying to fulfill some requirement, but certainly you have to adapt your code to match in the Vert.x spirit:
First you should better not block until a message is received in your websocket handler, keep in mind that everything is asynchrounous when it comes to Vert.x.
In order to sleep for some time, use the Vert.x way and not the Thread.sleep(delay), i.e. vertx.setTimer(...).
My basic problem is that I can only process 7000 messages from one of my queues (across all machines) in any 1 hour time period. I don't see a way to do this with camel or activemq, so I resorted to implementing my own route stopping/starting logic. I see a number of ways to do this, and I've tried a few of them (only to run into problems).
camelContext.stopRoute(route): This works in that messages stop being processed, but when I call camelContext.startRoute(route), it leaks a tcp connection which eventually causes the activemq server to hit its limit and die.
camelContext.suspendRoute(route): This also stops messages from being processed and does not leak connections, but it appears to kill active consumers (visible in the admin panel) that don't reactivate when I call camelContext.resumeRoute(route). I think that could eventually lead to no messages being processed off that queue at all, even if I resume.
implementing a custom RoutePolicy. To be fair, I haven't tried this yet, but it seems that it would fall prey to the same problems I had according to the pause method I chose above.
Is there a method for solving this problem that I haven't encountered yet?
Instead of stopping the route, I would recommend to use the Throttler EIP.
from("jms:queue:inbox")
.throttle(7000)
.timePeriodMillis(1000*60*60)
.to("log:result", "mock:result");
The above example will throttle messages received on jms:queue:inbox before being sent to mock:result ensuring that a maximum of 7000 messages are sent in any 1 hour window.
Alternatively, for more fine grained control you may define a throttling route policy as shown in Camel's throttling example:
<route routePolicyRef="myPolicy">
<from uri="jms:queue:inbox"/>
<transacted/>
<to uri="log:+++JMS +++?groupSize=100"/>
<to ref="foo"/>
</route>
The throttling police is defined as follows:
<bean id="myPolicy" class="org.apache.camel.impl.ThrottlingInflightRoutePolicy">
<property name="scope" value="Context"/>
<!-- when we hit > 20 inflight exchanges then kick in and suspend the routes -->
<property name="maxInflightExchanges" value="20"/>
<!-- when we hit lower than 10% of the max = 2 then kick in and resume the routes the default percentage is 70% but in this demo we want a low value -->
<property name="resumePercentOfMax" value="10"/>
<!-- output throttling activity at WARN level -->
<property name="loggingLevel" value="WARN"/>
</bean>
EDIT 1:
If you need a global throttling, then you may first let one consumer read the messages, throttling all messages as described above, then re-send them to another queue and let re-reading and processing them by >= 1 distributed consumers.
EDIT 2:
Alternatively, you may implement your own ThrottlingInflightRoutePolicy accessing a central database holding processing information. That way, you don't need a "single node master throttler". However, also the DB may be a single point of failure.
Peter's got the best answer(s), but I ended up extending ThrottlingInflightRoutePolicy, and there's no great explanation of how that works, so I thought I'd annotate this question a bit and show how I actually solved the problem.
public class MyRoutePolicy extends RoutePolicySupport implements CamelContextAware {
private CamelContext camelContext;
private final Lock lock = new ReentrantLock();
private ContextScopedEventNotifier eventNotifier;
#Override
public final void setCamelContext(final CamelContext camelContext) {
this.camelContext = camelContext;
}
#Override
public final CamelContext getCamelContext() {
return this.camelContext;
}
#Override
public final void onExchangeDone(final Route route, final Exchange exchange) {
throttle(route);
}
private void throttle(final Route route) {
// this works the best when this logic is executed when the exchange is done
Consumer consumer = route.getConsumer();
boolean stop = isRouteMarkedForSuspension(route.getId()) && ((JmsConsumer) route.getConsumer()).isStarted();
if (stop) {
try {
lock.lock();
stopConsumer(consumer);
} catch (Exception e) {
handleException(e);
} finally {
lock.unlock();
}
}
// reload size in case a race condition with too many at once being invoked
// so we need to ensure that we read the most current size and start the consumer if we are already to low
boolean start = !isRouteMarkedForSuspension(route.getId()) && ((JmsConsumer) route.getConsumer()).isSuspended();
if (start) {
try {
lock.lock();
startConsumer(consumer);
} catch (Exception e) {
handleException(e);
} finally {
lock.unlock();
}
}
}
#Override
protected final void doStart() throws Exception {
ObjectHelper.notNull(camelContext, "CamelContext", this);
eventNotifier = new ContextScopedEventNotifier();
// must start the notifier before it can be used
ServiceHelper.startService(eventNotifier);
// we are in context scope, so we need to use an event notifier to keep track
// when any exchanges is done on the camel context.
// This ensures we can trigger accordingly to context scope
camelContext.getManagementStrategy().addEventNotifier(eventNotifier);
}
#Override
protected final void doStop() throws Exception {
ObjectHelper.notNull(camelContext, "CamelContext", this);
camelContext.getManagementStrategy().removeEventNotifier(eventNotifier);
}
private class ContextScopedEventNotifier extends EventNotifierSupport {
#Override
public void notify(final EventObject event) throws Exception {
for (Route route : camelContext.getRoutes()) {
throttle(route);
}
}
#Override
public boolean isEnabled(final EventObject event) {
return event instanceof ExchangeCompletedEvent;
}
#Override
protected void doStart() throws Exception {
// noop
}
#Override
protected void doStop() throws Exception {
// noop
}
#Override
public String toString() {
return "ContextScopedEventNotifier";
}
}
}
So I added the RoutePolicy above to all of my routes, like this:
from(uri).routePolicy(routePolicy).process(runner);
MyRoutePolicy is a inner class and isRouteMarkedForSuspension is defined in the main class.
throttle is hit at two points:
after an exchange (message) is processed. This is useful for figuring out if the consumer should be paused.
on a notify event through the ContextScopedEventNotifier. This is useful for figuring out if the consumer should be resumed.
i want to do a throttling to a consumer of some queue in the activeMQ, in hornetq (of jboss, this is do it with annotations on the definition of the mdb Consumer). I can't find any similar in the documentation of activemq, the closest that i find was this
consumer.recvDelay 0 ms Pause consumer for recvDelay milliseconds with each message (allows consumer throttling).
from: http://activemq.apache.org/activemq-performance-module-users-manual.html
But there i can't find how i can do it in java.
Thanks in advance,
Regards.
EDIT: Here is the ActiveMQManager code and the consumer code:
public class ActiveMQManager {
private static ActiveMQConnectionFactory CONNECTION_FACTORY;
public static Connection CONNECTION;
public static Session SESSION;
public static Destination TEST_QUEUE;
public static void start() {
try {
CONNECTION_FACTORY = new ActiveMQConnectionFactory("vm://localhost");
CONNECTION = CONNECTION_FACTORY.createConnection();
CONNECTION.start();
SESSION = CONNECTION.createSession(false,
Session.CLIENT_ACKNOWLEDGE);
TestClient testClient = new TestClient();
TEST_QUEUE = SESSION.createQueue("TEST.QUEUE");
MessageConsumer testConsumer = SESSION.createConsumer(TEST_QUEUE);
test.setMessageListener(testClient);
} catch (Exception e) {
}
}
public static void stop() {
try {
// Clean up
SESSION.close();
CONNECTION.close();
} catch (JMSException e) {
log.error(e);
}
}
}
The consumer code is very simple (for this example):
public class TestConsumer implements MessageListener {
#Override
public void onMessage(Message message) {
//Do something with the message
}
}
this depends on the consumer technology being used...but here are a few options
you can manually introduce a delay in your consumer code (not an exact science but this will limit the throughput)
you can also control the number of threads that your consumer uses by setting maxConcurrentConsumers property of you JMS connection...that said, this won't throttle message throughput, just limit the level of concurrency being used by your consumer
better yet, you can set the exact number of messages to consume per time period using a throttler EIP implementation
for example, this is trivial using the Camel Throttler
from("activemq:queueA").throttle(10).to("activemq:queueB")
With ActiveMQ you can set the consumer prefetch limit: http://activemq.apache.org/what-is-the-prefetch-limit-for.html
To configure it, you can use the connection URL (most of the configurations can be done using the URL) or the Java API.
For more interesting parameters: http://activemq.apache.org/connection-configuration-uri.html
Take into account that Camel Throttler keeps the exchanges in-memory while they are blocked by the Throttler. So, if you have 100 queue consumers and the Server (exposing the SOAP service) is slow, you may have in-memory up to 100 exchanges!
Posted this question for this: Throttle consumption rate of all JMS consumers listening on an ActiveMQ queue