I am parsing a CSV file, splitting it and routing it through multiple processors in camel. There are two endpoints , one having erroneous data while other has validated data.
I need suggestion in aggregating the data.
Let's say the CSV file has 10 records out of which 6 reached one endpoint while 4 reached to another. How can I know if all 10 has completed from the file at each endpoint and move ahead of aggregator.
I need to create two files one with valid data and other with corrupt data from a single file.
Lets look at what the splitter returns.
According to the documentation on Camel 2.2. or older the splitter will by default return the last split message using your example this would probably be the last line to complete its processor so it might not be line 10 (using your example).
On Camel 2.3 and newer the splitter will by default return the original input message i.e. all 10 lines. This is the default behavior and you dont need to code anything for this to work. When the splitter is finished by default it will pass this message along to the next end point.
So if I was using the following DSL on Camel 2.3 or newer:
<camelContext trace="false" id="blueprintContext" xmlns="http://camel.apache.org/schema/blueprint">
<route id="splitExample">
<from uri="timer://myTimer?period=2000"/>
<setBody>
<simple>A\nB\nC</simple>
</setBody>
<log message="The message body before the splitter contains ${body}"/>
<split>
<tokenize token="\n"></tokenize>
<log message="Split line ${body}"/>
</split>
<log message="The message body after the splitter contains ${body}"/>
</route>
</camelContext>
The following would appear in the log:
INFO The message body before the splitter contains
A
B
C
INFO Split line A
INFO Split line B
INFO Split line C
INFO The message body after the splitter contains
A
B
C
As you can see camel by default combines the messages back into one after the splitter returns. To override this behavior you would need to implement your own aggregator. To do so create a class lets call it MyAggregationStrategy and make the class implement AggregationStrategy. I used the example in the apache documentation from here. example we will aggregate incoming bids and want to aggregate the highest bid.
private static class MyAggregationStrategy implements AggregationStrategy {
public Exchange aggregate(Exchange oldExchange, Exchange newExchange)
{
if (oldExchange == null)
{
// the first time we only have the new exchange so it wins the first round
return newExchange;
}
int oldPrice = oldExchange.getIn().getBody(Integer.class);
int newPrice = newExchange.getIn().getBody(Integer.class);
// return the "winner" that has the highest price
return newPrice > oldPrice ? newExchange : oldExchange;
}
}
After you have done this you then tell the splitter to use your aggregator by doing the following:
Spring/XML DSL:
<split strategyRef="MyAggregationStrategy ">
In Java:
from("direct:start")
// aggregated by header id and use our own strategy how to aggregate
.aggregate(new MyAggregationStrategy())
Hopefully this gives you enough insight about how the splitter works. In your case I would probably set a header value for each line indicating if it was successful or failed then I would use my customer aggregator to create a new message with failed and success grouped into two lists as the message body. One list with the failed and one list with the completed line items.
This new aggregated message can then be sent to a processor or another endpoint for further processing. For example you can then take the failed list and send that to a route which produces a file. The seda component can help a lot here.
Related
Using spring-camel, I have built a route that consumes from a JMS topic (with JMSReplyTo expected to be set for each input message), splits the message into smaller chunks, sends them to a REST processsor, then aggregates the answers and should produce an output message to the destination pointed by JMSReplyTo. Unfortunately, camel implicitly utilises the JMSReplyTo destination in one of the intermediate steps (producing an unmarshalled POJO).
We have a functional requirement to adapt JMSReplyTo in order to provide a request-reply messaging service.
I am able to read the JMSReplyTo header before ending the route and I am explicitly converting it to CamelJmsDestinationName, which successfully overrides the destination for JMS component and produces the message on the output topic. I am not sure if this is the best approach and the problem is that camel still utilises the JMSReplyTo on its own.
My RouteBuilder configuration is as follows:
from("jms:topic:T.INPUT")
.process(requestProcessor)
.unmarshal().json(JsonLibrary.Jackson, MyRequest.class)
.split(messageSplitter)
.process(restProcessor)
.aggregate(messagesAggregator)
.unmarshal().json(JsonLibrary.Jackson, BulkResponses.class)
.process(responseProcessor)
.to("jms:topic:recipientTopic");
T.INPUT is the name of the input topic, while recipientTopic is just a placeholder that will be replaced by CamelJmsDestinationName.
I'm not keen on using CamelJmsDestinationName and a sort of a mocked up topic name in route configuration so I'm open to find a better solution. It would be great if camel utilised the JMSReplyTo automatically to produce the output message to the output topic.
Currently, the problem is that camel produces an intermediate output on the JMSReplyTo topic BUT the output is an unmarshalled MyRequest object, which results in an exception saying "ClassNotFoundException: (package name).MyRequest", which is obvious since this is only a class used in my internal processing - I don't want to produce this to the output topic. It seems like Camel does implicitly use the JMSReplyTo destination between requestProcessor and messageSplitter processing... Why? What am I doing wrong? What are the best practices?
Use "disableReplyTo=true" in Endpoint. Camel will not try to use any reply option.
Refer: https://camel.apache.org/jms.html for more details
I have found the answer... this is absurdly easy but I haven't seen it anywhere in the documentation.
You just need to call .stop() to mark the route as completed, and Camel will reply the body you configured in the last step to the destination configured in ${header.JMSReplyTo}. It's that simple.
So you can do:
from("jms:my-queue")
.unmarshall().json(JsonLibrary.Jsonb, InboundMessage.class)
.bean(SomeProcessingBean.class)
....
.log(LoggingLevel.INFO, "Sending reply to: " + simple("${header.JMSReplyTo}").getExpression().toString())
.marshall().json(JsonLibrary.Jsonb, ReplyMessage.class)
.stop();
And you will receive reply.
I wonder why no one has found this before... Nothing when I search the doc or here.... I must be dumb, or the doc is incomplete...but I am not dumb, so.
I am writing a Camel integration that can consume an arbitrary number of queries and execute those against an arbitrary number of databases.
The route starts by getting all queries located in a folder and then uses a splitter to iterate over them in order:
from("quartz2:quartzInitializer?cron={{sync.cron}}")
.routeId("quartzInitializer")
.bean(QueryHandler.class, "getQueries")
.split(exchangeProperty(QueryHandler.Properties.QUERIES))
.setProperty(Properties.CURRENT_QUERY, simple("body"))
.to("direct:executeSingleQuery")
.end();
In the above snippet, the property QueryHandler.Properties.QUERIES contains two query file locations:
config/sql/1__select_stat_machine.sql
config/sql/2__select_stat_session.sql
Next, I send the location of the iterated query and construct a recipient list from it:
from("direct:executeSingleQuery")
.routeId("executeSingleQuery")
.bean(DataSourceHandler.class, "createEndpointsWithQuery")
.recipientList(exchangeProperty(DataSourceHandler.Properties.QUERY_RECIPIENTS))
.parallelProcessing()
.log(LoggingLevel.INFO, String.format("Calling ${in.header.%s}", Exchange.RECIPIENT_LIST_ENDPOINT));
In the above snippet, the parameter DataSourceHandler.Properties.QUERY_RECIPIENTS contains two recipients:
sql:file:config/sql/1__select_stat_machine.sql?dataSource=datasource3&outputHeader=resultset
sql:file:config/sql/1__select_stat_machine.sql?dataSource=datasource2&outputHeader=resultset
However, when I run this, only one of the recipients are called, in this case only datasource2, which was at index 1 in the list passed to the recipientList:
Calling sql://file:config/sql/1__select_stat_machine.sql?dataSource=datasource2&outputHeader=resultset
Calling sql://file:config/sql/2__select_stat_session.sql?dataSource=datasource2&outputHeader=resultset
I can't for the life of me figure out what I'm doing wrong. Am I missing an end() somewhere? Is my splitter at fault, or is it my recipient list?
.recipientList(exchangeProperty(...))
.log(LoggingLevel.INFO, String.format("Calling ${in.header.%s}", Exchange.RECIPIENT_LIST_ENDPOINT));
Your are putting the log statement in the wrong place.
Basically the way you have modelled your route is:
"Please send the messages to all recipients, and AFTER this, print a message". The fact is that after looping through the list of recipients, Camel variable holds the URI of the LAST recipient.
It is more obvious in Spring DSL:
What your Camel route is doing:
<recipientList>
<header>...</header>
</recipientList>
<log message="Done"/>
versus what you think Camel is doing:
<recipientList>
<header>...</header>
<log message="Done"/>
</recipientList>
My REST application will post data to a queue (Q1) on rabbitMQ. There's another separate application that will read from Q1, process the data and post the result back to Q2. My application will read the data from Q2 and return the result. Many clients will use these 2 queues so I generate a UUID and set it in the header so that I can listen on Q2 (the response topic). I will then query each incoming message and match the incoming UUID in the header to the one I generated when I posted to Q1.
from("direct:test")
.choice().when(isValid)
.bean(FOOProcessor.class, "setFooQuery")
.to(FOO_REQUEST_QUEUE).log(LoggingLevel.INFO, "body=${in.body}")
.otherwise()
.setBody(constant("error"))
.setHeader(Exchange.HTTP_RESPONSE_CODE, constant(400)).log(LoggingLevel.INFO, "body=${in.body}")
.to("direct:error");
from(FOO_RESPONSE_QUEUE)
.unmarshal(new JacksonDataFormat(JsonNode.class))
.bean(FooProcessor.class, "setFooResponse")
.to("direct:end");
from("direct:error").log(LoggingLevel.DEBUG, "end");
from("direct:end").log(LoggingLevel.DEBUG, "end");
The trouble is the 2 "from" statements - they create to separate Camel exchanges/contexts and I can't get the original UUID. Any suggestions?
I solved this by using a processor that had a route builder embedded in it (with its own producer and consumer).
The processor provided a reference to the main exchange from this
process(final Exchange exchange)
We're using Spring Integration 4.2.0. We have a flow that uses a Message Router and have a desire to be able to log where a message was routed to (actual Destination name and ideally Destination type along with the raw payload). In our case our routers have output channels which have JmsSendingMessageHandler's as endpoints.
What we would like to see is something like this in our logs:
[INFO ] message routed to [amq | queue://QUEUE1] : This is a message!
[INFO ] message routed to [wmq | queue://QUEUE2] : This is also a message!
[INFO ] message routed to [ems | queue://QUEUE3] : This is also a message!
[INFO ] message routed to [wmq | topic://TOPIC1] : This is also a message!
The router config is similar to this:
<int:router id="messageRouter"
input-channel="inputChannel"
resolution-required="false"
ref="messageRouterServiceImpl"
method="route"
default-output-channel="unroutedChannel">
<int:mapping value="channelAlias1" channel="channel1" />
<int:mapping value="channelAlias2" channel="channel2" />
<int:mapping value="channelAlias3" channel="channel3" />
<int:mapping value="routerErrorChannel" channel="routerErrorChannel"/>
<int:mapping value="nullChannel" channel="nullChannel"/>
</int:router>
I have a solution for achieving this but I'll admit it is a bit ugly as it queries the Spring ApplicationContext then uses reflection to ultimately obtain the Destination's name.
Alternatively I suppose I could put a logger at the front of every channel that the router outputs to but was trying to avoid repeatedly having to remember to do this for every flow that we use a router in.
I'm wondering if anyone has suggestions for a cleaner way of doing this. I can share my code if you'd like. Perhaps Spring Integration Java DSL would help with this?
The one of point where you can hook is <int:wire-tap> - a global ChannelInterceptor for the particular channels by the pattern for their name.
This WireTap may send messages to the <int:logging-channel-adapter> or any other custom service to log via desired way or do anything else.
Another good out-of-the-box feature for you is <int:message-history>. With that you will have a path how message has traveled through your flow, including the routing logic. You can find it as a part of the MessageHeaders.
If I understand your use case correctly, you want
router to log this information after successfully sending the message to expected destination.
You do not want to put loggers, as it will be forced to have for every new flow attached to router
One approach which I can think of is-
1- Extend MethodInvokingRouter for your custom router implementation.
2- Override handleMessageInternal method from AbstractMessageRouter class.
Here is the code snippet-
public class CustomRouter extends MethodInvokingRouter {
Logger log = Logger.getLogger(CustomRouter.class);
Map<Message<?>,String> m = new ConcurrentHashMap<>();
public CustomRouter(Object object, Method method) {
super(object, method);
}
public CustomRouter(Object object, String methodName) {
super(object, methodName);
}
public CustomRouter(Object object) {
super(object);
}
public String route(Message<?> message) {
String destinationName = null;
/*
* Business logic to get destination name
*/
destinationName = "DERIVED VALUE AS PER THE ABOVE BUSINESS LOGIC";
//NOTE:- here also we can log (optimistic way of saying that messages are being routed to...), but at this point of time we are not sure whether the returned string value will get resolved to a message channel and message will be routed to the destination.
//Put that name to map so that it can be extracted later on in handleMessageInternal method
m.put(message, destinationName); // O(1) complexity
return destinationName;
}
#Override
protected void handleMessageInternal(Message<?> message) {
super.handleMessageInternal(message);
//At this point we are quit sure that channel has been resolved and message has been sent to destination
/*
* get the destination name returned from route method from populated map
*
* As at this point we know whatever return value was (from route method), there exists a message channel.
*
*/
String key = m.get(message);
// get the key-value pair for channel mapping
Map<String,String> mappedChannelMap = super.getChannelMappings();
// get destination name where message is routed
String destinationName = mappedChannelMap.get(key); // O(1) complexity
//Now log to a file as per the requirement
log.info("message routed to "+destinationName+" Message is- "+message.getPayload().toString());
}
}
I haven't tried this piece of code, there may exist some improvement. What's your thought...
I have one problem, and don't know how to solve it using camel. I searched for related EIP in camel documentation, but without results.
Now I have simple route:
<route id="routeId">
<from uri="Source1"/>
<from uri="Source2"/>
<to uri="Destination"/>
</route>
Both sources sends JMS messages to Destination and at some point when Source finish its job it send specific end message, with some flag. What I need to do is to collect or count those end messages and send single end message to destination when I receive end messages from both sources. Only when i receive two end messages (imagine that its just simple message with some header flag) then i should send single one to destination.
Sorry if problem explanation isn't clear enough.
Thanks in advance.
the Camel aggregator and filter patterns can be used for this scenario...
use a filter to detect "end" messages and route them through an aggregator
use a custom aggregation strategy to build up the single end message with a count
use a custom completion predicate to trigger the completion message
something like this...
from("source1").to("direct:aggregateRoute");
from("source2").to("direct:aggregateRoute");
from("direct:aggregateRoute")
.filter(header("isEndMessage").isEqualTo("true"))
.aggregate(constant(true), new MyAggregationStrategy())
.completionPredicate(new MyCompletionStrategy())
.to("destination");
If you just want to pick from multiple inputs and does not want to perform any modification on the incoming message,
you can do something like this:
from("URI1", "URI2", "URI3").to("DestinationUri");
for more info check this link it helped me