After long time of research i still have no acceptable solution for my purposes.
Intend:
I'm trying to have multiple Endpoints/Components which are generating exchanges with proprietary content. (1st row 1st col in picture)
Thus exchanges should be normalized by a normalizer/translator (1st row 2nd col in picture)
Finally this normalized data should be processed. (Saved to database, did some aggregations, calculating some results).
After a result is generated, a completely new exchange should be generated and populated to routes, where this processor is located in the "from" statement. (2nd Row) -> Data should be translated to proprietary data -> send to endpoint
Questions & Preliminary Findings
A Processor could not be located in the "from" statement directly. I
only found some workarounds in the manner " from
(direct:somemessages)" which is created in the processor by
to(direct:somemessages)
The Processor could be programmes as a Component with an endpoint.
And the asynchronus messages could be published with the
corresponding Consumer
Example Route Ingoing (1st row)
<route id="bar">
<from uri="mqtt"/>
<to uri="TranslateMQTT2MyModel"/>
<to uri="ProcessData"/>
</route>
Example Route Outgoing (i would like to have) (2nd row)
<route id="out">
<from uri="ProcessData"/>
<to uri="TranslateMyModel2MQTT"/>
<to uri="mqtt"/>
</route>
So how I can achieve my purposes? Is one of my two outcomes a good solution?
Example of schematic workflow
You could write a "ProcessData" component that is a consumer, but I don't think you need to do that.
I think you can use the asynchronous capability of the SEDA component, so I think what you're looking for is something like:
<route id="bar">
<from uri="mqtt"/>
<to uri="TranslateMQTT2MyModel"/>
<to uri="seda:processAndReply"/>
</route>
<route id="out">
<from uri="seda:processAndReply" />
<to uri="ProcessData"/>
<to uri="TranslateMyModel2MQTT"/>
<to uri="mqtt"/>
</route>
The SEDA component works similarly to direct as you described, but direct is synchronous and SEDA is asynchronous. (Also note the VM and Direct-VM components if you need to link between Camel Contexts with the same JVM).
Related
Here is a simplified ftp polling mechanism.
<camelContext id="Fetcher" xmlns="http://camel.apache.org/schema/blueprint">
<redeliveryPolicyProfile id="redeliveryPolicy"
redeliveryDelay="10000"
maximumRedeliveries="-1" />
<camel:route id="fetchFiles">
<camel:from uri="ftp://10.20.30.40/From?username=user&password=RAW({{password}})&delay=3000" />
<camel:to uri="log:input?showAll=true&level=INFO"/>
<camel:to uri="file://incomingDirectory" />
<onException redeliveryPolicyRef="msRedeliveryPolicy">
<exception>java.lang.Exception</exception>
<redeliveryPolicy logRetryAttempted="true" retryAttemptedLogLevel="WARN"/>
</onException>
</camel:route>
</camelContext>
What do you think happens on failure? (Delay is 3 seconds, and
redeliveryDelay is 10 seconds.)
Answer: It polls every 3 seconds, forever.
So let's look at the docs. Maybe I need this
"repeatCount (scheduler)"
Specifies a maximum limit of number of fires. So if you set it to 1, the scheduler will only fire once. If you set it to 5, it will only fire five times. A value of zero or negative means fire forever.
Default: 0
Nope, it's not even a valid parameter. So why's it in the docs?
Unknown parameters=[{repeatCount=5}]
Ok, so I suppose every 3 seconds it polls. So how do I tell camel to stop that? Let's try set 'handled' to true?
<onException redeliveryPolicyRef="msRedeliveryPolicy">
<exception>java.lang.Exception</exception>
<redeliveryPolicy logRetryAttempted="true" retryAttemptedLogLevel="WARN"/>
<handled><constant>true</constant></handled>
</onException>
No luck. Still 3 seconds. It's clearly not even getting to the redelivery part.
What's the secret?
The fact is errors happen in from endpoint are not handled by user defined route (i.e. fetchFiles in above setup). So, onException and redeliveryPolicy are not involved as they only affect stuff belongs to user defined route.
To control the behavior of consumer defined in from endpoint, the obvious way is to use the option exist in that component. As suggested by #Screwtape, use backoffErrorThreshold and backoffMultplier for your case.
Why parameter repeatCount exist in doc, but is invalid to use? It probably does not exist in your camel version and Camel document writer forget to mark the first exist version in the doc.
I have the following route...
<route id="VM01_spit_products">
<from uri="direct:processXML" />
<split>
<method ref="CamelSplitOnKey" method="splitIntoBatches" />
<to uri="vm:processXMLSplit" />
</split>
</route>
<route id="VM01_processXML">
<from uri="vm:processXMLSplit?concurrentConsumers=15" />
<bean ref="Builder" method="createXMLFile" />
<to uri="{{ChangeReceiver}}" />
</route>
I was expecting with the use of VM or SEDA to mean that if the spliter is producing 5 messages, then one of the 15 threads I have defined would pick up each of these messages. When I debug into the Builder class, I can see that the messages are being picked up sequentially.
I see the same if I an using VM or SEDA.
Can someone suggest where I'm going wrong?
Notes:
Camel 2.6 due to JDK 1.5
New info.
I've added this code into my Builder.java
SedaEndpoint seda = (SedaEndpoint) camelContext.getEndpoint("seda:processXMLSplit");
int size = seda.getExchanges().size();
System.out.println("size ["+size+"]");
This prints a size of 0 each time.
This makes me think that the split isn't queuing up the messages as I expect.
Even if you have defined your vm consumer to have 15 threads it doesn't affect how your Split is working. By default Split is working sequentially so you must configure your Split to use parallelProcessing in order to get the result you want. See further in Splitter ParallelProcessing. Note also #Itsallas comment that you might need your vm endpoint configured with the same parameters.
We are in the process of building a Java application with Camel to transfer files between two FTP locations. Is there a way to get a notification on the successful transfer of a file? We are not allowed to use a JMS solution for building the application.
I hope you could create another route and have seda/vm as the endpoint.This endpoint needs to be called after the ftp endpoint.
<route id="MainRoute">
<from uri="ftp:RemoteLocation"/>
<from uri="seda:Retry"/>
<to uri="seda:MyLog"/>
<!--Your Main Processing logic -->
</route>
<route id="Notification-processor">
<from uri="seda:MyLog"/>
<!--Your Logging/Notification Processing logic -->
</route>
In the above scenario of Notification-processor you can have your custom notificaiton/log activity. This is your custom notification logic. If you need to notify for anomalies you can have a to endpoint in the Notification-processor for sending the notification.
You need to write logic to check if the message is complete if not you can have a bean called in the Notification-processor which can have dynamic route to extract the specific file form the ftp location and reprocess it. Like below
<route id="Notification-processor">
<from uri="seda:MyLog"/>
<!--Anomaly checker -->
<to uri="seda:Retry"/>
<!--Your Logging/Notification Processing logic -->
</route>
I have one problem, and don't know how to solve it using camel. I searched for related EIP in camel documentation, but without results.
Now I have simple route:
<route id="routeId">
<from uri="Source1"/>
<from uri="Source2"/>
<to uri="Destination"/>
</route>
Both sources sends JMS messages to Destination and at some point when Source finish its job it send specific end message, with some flag. What I need to do is to collect or count those end messages and send single end message to destination when I receive end messages from both sources. Only when i receive two end messages (imagine that its just simple message with some header flag) then i should send single one to destination.
Sorry if problem explanation isn't clear enough.
Thanks in advance.
the Camel aggregator and filter patterns can be used for this scenario...
use a filter to detect "end" messages and route them through an aggregator
use a custom aggregation strategy to build up the single end message with a count
use a custom completion predicate to trigger the completion message
something like this...
from("source1").to("direct:aggregateRoute");
from("source2").to("direct:aggregateRoute");
from("direct:aggregateRoute")
.filter(header("isEndMessage").isEqualTo("true"))
.aggregate(constant(true), new MyAggregationStrategy())
.completionPredicate(new MyCompletionStrategy())
.to("destination");
If you just want to pick from multiple inputs and does not want to perform any modification on the incoming message,
you can do something like this:
from("URI1", "URI2", "URI3").to("DestinationUri");
for more info check this link it helped me
I'm using Camel (2.11.0) to try and achieve the following functionality:
If a file exists at a certain location, copy it to another location and then begin processing it
If no such file exists, then I don't want the file consumer/poller to block; I just want processing to continue to a direct:cleanup route
I only want the file to be polled once!
Here's what I have so far (using Spring XML):
<camelContext id="my-camel-context" xmlns="http://camel.apache.org/schema/spring">
<route id="my-route
<from uri="file:///home/myUser/myApp/fizz?include=buzz_.*txt"/>
<choice>
<when>
<!-- If the body is empty/NULL, then there was no file. Send to cleanup route. -->
<simple>${body} == null</simple>
<to uri="direct:cleanup" />
</when>
<otherwise>
<!-- Otherwise we have a file. Copy it to the parent directory, and then continue processing. -->
<to uri="file:///home/myUser/myApp" />
</otherwise>
</choice>
<!-- We should only get here if a file existed and we've already copied it to the parent directory. -->
<to uri="bean:shouldOnlyGetHereIfFileExists?method=doSomething" />
</route>
<!--
Other routes defined down here, including one with a "direct:cleanup" endpoint.
-->
</camelContext>
With the above configuration, if there is no file at /home/myUser/myApp/fizz, then Camel just waits/blocks until there is one. Instead, I want it to just give up and move on to direct:cleanup.
And if there is a file, I see it getting processed inside the shouldOnlyGetHereIfFileExists bean, but I do not see it getting copied to /home/myUser/myApp; so it's almost as if the <otherwise> element is being skipped/ignored altogether!
Any ideas? Thanks in advance!
Try this setting, and tune your polling interval to suit:
From Camel File Component docs:
sendEmptyMessageWhenIdle
default =false
Camel 2.9: If the polling consumer did not poll any files, you can enable this option to send an empty message (no body) instead.
Regarding writing the file, add a log statement inside the <otherwise> to ensure it's being executed. If so, check file / folder permissions, etc.
Good luck.
One error i faced while I tried using the condition:
<simple>${body} != null</simple>
was it always returns true.
Please go through the below link:
http://camel.465427.n5.nabble.com/choice-when-check-BodyType-null-Body-null-td4259599.html
It may help you.
This is very old, but if anyone finds this, you can poll only once with
"?repeatCount=1"
I know the question was done almost 4 years ago but I had exaclty the same problem yesterday.
So I will let my answer here, maybe it will help another person.
I am using Camel, version 3.10.0
To make it work exactly as described in the question:
If a file exists at a certain location, copy it to another location and then begin processing it
If no such file exists, then I don't want the file consumer/poller to block; I just want processing to continue to a direct:cleanup route
ONLY want the file to be polled once!
Using ${body} == null
The configuration that we need are:
sendEmptyMessageWhenIdle=true //Will send a empty body when Idle
maxMessagesPerPoll=1 //Max files that it will take at once
repeatCount=1 //How many times will execute the Pool (above)
greedy=true // If the last pool executed with files, it will
execute one more time
XML:
<camel:endpoint id="" uri="file:DIRECTORY?sendEmptyMessageWhenIdle=true&initialDelay=100&maxMessagesPerPoll=1&repeatCount=1&greedy=false" />