My REST application will post data to a queue (Q1) on rabbitMQ. There's another separate application that will read from Q1, process the data and post the result back to Q2. My application will read the data from Q2 and return the result. Many clients will use these 2 queues so I generate a UUID and set it in the header so that I can listen on Q2 (the response topic). I will then query each incoming message and match the incoming UUID in the header to the one I generated when I posted to Q1.
from("direct:test")
.choice().when(isValid)
.bean(FOOProcessor.class, "setFooQuery")
.to(FOO_REQUEST_QUEUE).log(LoggingLevel.INFO, "body=${in.body}")
.otherwise()
.setBody(constant("error"))
.setHeader(Exchange.HTTP_RESPONSE_CODE, constant(400)).log(LoggingLevel.INFO, "body=${in.body}")
.to("direct:error");
from(FOO_RESPONSE_QUEUE)
.unmarshal(new JacksonDataFormat(JsonNode.class))
.bean(FooProcessor.class, "setFooResponse")
.to("direct:end");
from("direct:error").log(LoggingLevel.DEBUG, "end");
from("direct:end").log(LoggingLevel.DEBUG, "end");
The trouble is the 2 "from" statements - they create to separate Camel exchanges/contexts and I can't get the original UUID. Any suggestions?
I solved this by using a processor that had a route builder embedded in it (with its own producer and consumer).
The processor provided a reference to the main exchange from this
process(final Exchange exchange)
Related
Using spring-camel, I have built a route that consumes from a JMS topic (with JMSReplyTo expected to be set for each input message), splits the message into smaller chunks, sends them to a REST processsor, then aggregates the answers and should produce an output message to the destination pointed by JMSReplyTo. Unfortunately, camel implicitly utilises the JMSReplyTo destination in one of the intermediate steps (producing an unmarshalled POJO).
We have a functional requirement to adapt JMSReplyTo in order to provide a request-reply messaging service.
I am able to read the JMSReplyTo header before ending the route and I am explicitly converting it to CamelJmsDestinationName, which successfully overrides the destination for JMS component and produces the message on the output topic. I am not sure if this is the best approach and the problem is that camel still utilises the JMSReplyTo on its own.
My RouteBuilder configuration is as follows:
from("jms:topic:T.INPUT")
.process(requestProcessor)
.unmarshal().json(JsonLibrary.Jackson, MyRequest.class)
.split(messageSplitter)
.process(restProcessor)
.aggregate(messagesAggregator)
.unmarshal().json(JsonLibrary.Jackson, BulkResponses.class)
.process(responseProcessor)
.to("jms:topic:recipientTopic");
T.INPUT is the name of the input topic, while recipientTopic is just a placeholder that will be replaced by CamelJmsDestinationName.
I'm not keen on using CamelJmsDestinationName and a sort of a mocked up topic name in route configuration so I'm open to find a better solution. It would be great if camel utilised the JMSReplyTo automatically to produce the output message to the output topic.
Currently, the problem is that camel produces an intermediate output on the JMSReplyTo topic BUT the output is an unmarshalled MyRequest object, which results in an exception saying "ClassNotFoundException: (package name).MyRequest", which is obvious since this is only a class used in my internal processing - I don't want to produce this to the output topic. It seems like Camel does implicitly use the JMSReplyTo destination between requestProcessor and messageSplitter processing... Why? What am I doing wrong? What are the best practices?
Use "disableReplyTo=true" in Endpoint. Camel will not try to use any reply option.
Refer: https://camel.apache.org/jms.html for more details
I have found the answer... this is absurdly easy but I haven't seen it anywhere in the documentation.
You just need to call .stop() to mark the route as completed, and Camel will reply the body you configured in the last step to the destination configured in ${header.JMSReplyTo}. It's that simple.
So you can do:
from("jms:my-queue")
.unmarshall().json(JsonLibrary.Jsonb, InboundMessage.class)
.bean(SomeProcessingBean.class)
....
.log(LoggingLevel.INFO, "Sending reply to: " + simple("${header.JMSReplyTo}").getExpression().toString())
.marshall().json(JsonLibrary.Jsonb, ReplyMessage.class)
.stop();
And you will receive reply.
I wonder why no one has found this before... Nothing when I search the doc or here.... I must be dumb, or the doc is incomplete...but I am not dumb, so.
I'm new to RabbitMQ and want to implement asynchronous messaging of SAGA with RabbitMQ.So I used RPC example of RabbitMQ to do the task. I've one orchestrator ( RPCClient) and multiple microservices ( RPCServer). Orchestrator uses unique queues to command microservices.And each microservice uses a common queue ( Reply_ Queue) to reply orchestrator. To keep log I want to get notifications in orchestrator side, when any microservice is down for any configurable time.
I read about consumer cancellation,but it only works when I delete the queue.How to get notifications in JAVA with keeping queue messages? And is it correct way to implement saga asynchronous messaging?
To implement a reliable RPC is hard, I can't give a detail guide about how to do this. If we ignore same special failure situation, I can give a simple workaround:
First, we assume that RPCClient never fail, RPCServer may fail anytime.
RPCClient need to know which request is timeout, so it can send request message with a TTL. After RPCServer receive request message and send response message, it should ACK the request message.
If RPCServer:
has failed before consume request message
OR
has failed before send response message
The request message will be republish to Dead Letter Exchange, so RPCClient can consume to some queue binded with that exchange, it can know which request is timeout.
I am building a system that will receive messages via a Message broker (Currently, JMS) from different systems. All the messages from all the senders systems have a deviceId and there is no order in the reception of the message.
For instance, system A can send a message with deviceId=1 and system b be can send a message with deviceId=2.
My goal is not to start processing of the messages concerning the same deviceId unless I got all the message from all the senders with the same deviceId.
For example, if I have 3 systems A, B and C sending messages to my system :
System A sends messageA1 with deviceId=1
System B sends messageB1 with deviceId=1
System C sends messageC1 with deviceId=3
System C sends messageC2 with deviceId=1 <--- here I should start processing of messageA1, messageB1 and messageC2 because they are having the same deviceID 1.
Should this problem be resolved by using some sync mechanism in my system , by the message broker or an integration framework like spring-integration/apache camel ?
A similar solution with the Aggregator (what #Artem Bilan mentioned) can also be implemented in Camel with a custom AggregationStrategy and with controlling the Aggregator completion by using the Exchange.AGGREGATION_COMPLETE_CURRENT_GROUP property.
The following might be a good starting point. (You can find the sample project with tests here)
Route:
from("direct:start")
.log(LoggingLevel.INFO, "Received ${headers.system}${headers.deviceId}")
.aggregate(header("deviceId"), new SignalAggregationStrategy(3))
.log(LoggingLevel.INFO, "Signaled body: ${body}")
.to("direct:result");
SignalAggregationStrategy.java
public class SignalAggregationStrategy extends GroupedExchangeAggregationStrategy implements Predicate {
private int numberOfSystems;
public SignalAggregationStrategy(int numberOfSystems) {
this.numberOfSystems = numberOfSystems;
}
#Override
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
Exchange exchange = super.aggregate(oldExchange, newExchange);
List<Exchange> aggregatedExchanges = exchange.getProperty("CamelGroupedExchange", List.class);
// Complete aggregation if we have "numberOfSystems" (currently 3) different messages (where "system" headers are different)
// https://github.com/apache/camel/blob/master/camel-core/src/main/docs/eips/aggregate-eip.adoc#completing-current-group-decided-from-the-aggregationstrategy
if (numberOfSystems == aggregatedExchanges.stream().map(e -> e.getIn().getHeader("system", String.class)).distinct().count()) {
exchange.setProperty(Exchange.AGGREGATION_COMPLETE_CURRENT_GROUP, true);
}
return exchange;
}
#Override
public boolean matches(Exchange exchange) {
// make it infinite (4th bullet point # https://github.com/apache/camel/blob/master/camel-core/src/main/docs/eips/aggregate-eip.adoc#about-completion)
return false;
}
}
Hope it helps!
You can do this in Apache Camel using a caching component. I think there is the EHCache component.
Essentially:
You receive a message with a given deviceId say deviceId1.
You look up in your cache to see which messages have been received for deviceId1.
As long as you have not received all three you add the current system/message to the cache.
Once all messages are there you process and clear the cache.
You could then off course route each incoming message to a specific deviceId based queue for temporary storage. This can be JMS, ActiveMQ or something similar.
Spring Integration provides component for exactly this kind of tasks - do not emit until the whole group is collected. And it's name an Aggregator. Your deviceId is definitely a correlationKey. The releaseStrategy really may be based on the number of systems - how much deviceId1 messages you are waiting before proceed to the next step.
When I sent messages in a Camel Context Component to its endpoint, I have to wait for a response message with acknowledgement. If no response is received within timeout time, an exception shall be thrown back to the camel route.
I tried to implement it the following way:
I used a multicast to generate a timeout response while the original message is sent to the endpoint. The timeout response is delayed and if no response is received after this timeout, a timeout exception shall be thrown back on the route.
So I have the following route:
private final String internalRespUri = "direct:internal_resp";
private final String internalRespTimeout = "seda:internaltimeout";
#Override
public void configure() {
SendController send_controller = new SendController();
TimeoutResponse resp = new TimeoutResponse();
from(Endpoints.MESSAGE_IN.direct())
.errorHandler(noErrorHandler())
.routeId(Endpoints.MESSAGE_IN.atsm())
.log("Incoming message at segment in")
.process(send_controller)
.log("Message after send controller")
.multicast().parallelProcessing()
.log("After wiretap")
.to(internalRespTimeout, Endpoints.SEGMENT_OUT.direct());
from(internalRespTimeout)
.errorHandler(noErrorHandler())
.routeId(internalRespTimeout)
.log("begin response route")
.log("timeout response route")
.process(resp)
.log("modify message to response")
.delay(1000)
.log("after delay")
.to(internalRespUri);
from(Endpoints.SEGMENT_IN.seda())
.routeId(Endpoints.SEGMENT_IN.atsm())
.to(internalRespUri);
from(internalRespUri)
.errorHandler(noErrorHandler())
.routeId(internalRespUri)
.log("after response gathering point")
.choice()
.when(header(HeaderKeys.TYPE.key()).isEqualTo(UserMessageType.RESP.toString()))
.log("process responses")
.process(send_controller)
.otherwise()
.log("no response")
.to(Endpoints.MESSAGE_OUT.direct());
}
The problem is that the exception thrown in the SendController is not propagated over the SEDA endpoint internalRespTimeout.
If I use a direct endpoint instead it works, but then I have another problem:
The delay blocks the route while a received response message from endpoint Endpoints.SEGMENT_IN.seda() may not be transmitted.
Are SEDA endpoint generally not able to propagate exceptions?
How can I achieve a solution to my problem?
Thanks,
Sven
I have an idea:
Instead of throwing an exception, I possibly could use transactions for timeout.
Could this work?
I am currently not aware of a way to propagate and exception back over a SEDA endpoint in camel. The way the error handling works is based on channels between endpoints. When you use a SEDA endpoint the code will keep processing and not wait for the code since it will keep processing. I am having a bit of trouble understanding what you would like to accomplish, but I will list some similar alternatives you might be able to use.
-The first is to use a route level error handler in your SEDA based route and store the exception using a unique Id that you can lookup later.
-The second is to pass the data into a Java Bean where you have full control of what you are doing and could even consider something like using a Guava's Futures to run the code asynchronously while doing other tasks.
If you can explain what you are trying to accomplish a bit better I might be able to make a clearer suggestion.
RabbitMQ's Channel#basicConsume method gives us the following arguments:
channel.basicConsume(queueName, autoAck, consumerTag, noLocal,
exclusive, arguments, callback);
Giving us the ability to tell RabbitMQ exactly which queue we want to consume from.
But Channel#basicPublish has no such equivalency:
channel.basicPublish(exchangeName, routingKey, mandatory, immediateFlag,
basicProperties, messageAsBytes);
Why can't I specify the queue to publish to here?!? How do I get a Channel publishing to, say, a queue named logging? Thanks in advance!
To expand on #Tien Nguyen's answer, there is a "cheat" in RabbitMQ that effectively lets you publish directly to a queue. Each queue is automatically bound to the AMQP default exchange, with the queue's name as the routing key. The default exchange is also known as the "nameless exchange" - ie its name is the empty string. So if you publish to the exchange named "" with routing key equal to your queue's name, the message will go to just that queue. It is going through an exchange as #John said, it's just not one that you need to declare or bind yourself.
I don't have the Java client handy to try this code, but it should work.
channel.basicPublish("", myQueueName, false, false, null, myMessageAsBytes);
That said, this is mostly contrary to the spirit of how RabbitMQ works. For normal application flow you should declare and bind exchanges. But for exceptional cases the "cheat" can be useful. For example, I believe this is how the Rabbit Admin Console allows you to manually publish messages to a queue without all the ceremony of creating and binding exchanges.
Basically queues can be binded to an exchange based on routingKeys.
Assume that you have 3 different publishers.
Publisher1 sending message to exchange with routingKey "events"
Publisher2 sending message to exchange with routingKey "tasks"
Publisher3 sending message to exchange with routingKey "jobs"
You can have a consumer that consumes only messages with specific routhingKey.
For example in order to have a consumer for "events" messages you declare like this
channel.queueBind(queueName, exchangeName, "events");
If you want to consume all the messages coming to the exchange you give the routing as '#'
So in short what i can say is,
1. Messages will be published to an exchange.
2. Queues will be bound to exchange based on routingKeys.
3. RabbitMQ will forward messages with matching routing keys to the corresponding queues.
Please see the tutorial - http://www.rabbitmq.com/tutorials/tutorial-three-java.html
The core idea in the messaging model in RabbitMQ is that the producer never sends any messages directly to a queue. Actually, quite often the producer doesn't even know if a message will be delivered to any queue at all. Instead, the producer can only send messages to an exchange
please try this:
channel.basicPublish("", yourQueueName, null,
message.getBytes((Charset.forName("UTF-8"))));
It worked for my project.