Hope you have some spare minutes for my question. For the last couple of days I was reading about Camel and managed to setup everything up and running. Now, I have bumped on a tricky part :). Basically, I define a route using Java in runtime and put the route configuration in DB. Routes are working and messages are flowing from one side to another. But, when an exception occurs I would like to know where the exception has occurred (between which route endpoints), store the current exchange body (in-flight message that is useful for further processing) in the DB, update the message by the user and then retry the route execution from the point of exception (where it failed). Lets say that all route steps are idempotent.
My solution:
Make a Interceptor for the route
Granulate the route in as much as possible parts (each new step is a new route)
Between each step update the DB with the current and future step and current exchange body
If exception occurs, store the message in DB, update it using an editor
Send a message to a next route point (taken from DB) using ProducerTemplate
What do you think about this ? Is it doable or Camel cannot support me with this approach ?
Thank you for patience and your time.
Hope I was clear enough.
You can use Camel's tracer component. A detailed example meeting your needs is already available on camel's site : http://camel.apache.org/tracer-example.html
You should use onException() clause to tackle this. For eg :
public void configure() throws Exception{
//This is a global onException definition and will work for all routes defined in this calss
onException().process(new Processor(){
public void process(Exchang arg0){
Exception e = arg0.getProperty(Exchange.EXCEPTION_CAUGHT,Exception.class);
//get message and other properties that you are interested in
db.saveOrUpdate(/*Pass your custom object here*/);
}
}).handled(true);
from("direct:route1")
//some processing
.to("jms:route1");
from("direct:route2")
//some processing
.to("http://route2");
}
You might need to consult exact details at apache camel site, since i just wrote this code here.
Related
I am using Camel Timer component to read blobs from Azure storage container. A route is created which will poll for blobs every 10secs and is processed by the CloudBlobProcessor.
from("timer://testRoute?fixedRate=true&period=10s")
.to("azure-blob://storageAccountName/storageContainerName?credentials=#credentials")
.to(CloudBlobProcessor)
.to("mock:result");
I want to write a testcase by creating a mock endpoint something like this
MockEndpoint timerMockEndpoint = context.getEndpoint("timer://testRoute?fixedRate=true&period=10s", MockEndpoint.class);
But, I receive a below exception while creating the above mock endpoint.
java.lang.IllegalArgumentException: The endpoint is not of type:
class org.apache.camel.component.mock.MockEndpoint but is: org.apache.camel.component.timer.TimerEndpoint
Below is the code where I am trying to skip sending to the original endpoint
#Override
protected RoutesBuilder createRouteBuilder() throws Exception {
return new AdviceWithRouteBuilder() {
#Override
public void configure() throws Exception {
interceptSendToEndpoint("timer://testRoute?fixedRate=true&period=10s").skipSendToOriginalEndpoint()
.log("Original Batch Endpoint skipped")
.to("azure-blob://*")
.to(CloudBlobProcessor).to("mock:result");
from("timer://testRoute?fixedRate=true&period=10s").to("mock:result");
}
};
}
What I understand, we're trying to solve two different problems here:
MockEndpoint != TimerEndpoint
Interceptions
Answer to the first one is simple: MockEndpoints follow syntax mock:name. TimerEndpoint is a different endpoint and a totally different object. I don't know what you're aiming to do with the MockEndpoint here, but we just can't technically have a TimerEndpoint object as a MockEndpoint object. Why? Because that's how object oriented programming and Java work.
Let's take a look on the second problem. I've less than a year experience with Camel and I've only used interception once last year, but I hope I can guide you to some helpful direction.
The point of interception is to say "don't do that, do this instead". In this use case, it seems that we're only trying to skip sending a request to azure-blob endpoint. I'd try intercepting azure-blob://storageAccountName/storageContainerName?credentials=#credentials.
So instead of your interception, I'd try writing an interception like this:
interceptSendToEndpoint("azure-blob://storageAccountName/storageContainerName?credentials=#credentials")
.skipSendToOriginalEndpoint()
.log("Intercepted!");
In this case, instead of sending the request to azure-blob we intercept that request. We're telling Camel to skip the send to original endpoint, which means nothing will be sent to azure-blob://storageAccountName/storageContainerName?credentials=#credentials. Instead, we'll log "Intercepted!".
I am trying to get familiar with EIP and Apache Camel and I have a use case that I am not quite sure how can be expressed or implemented using Camel
Use case:
Imagine you have designed an integration solution that takes files from an ftp, does some processing and uploads it to a queue. You chose Apache Camel to implement this solution and your route in Java DSL looks something like this:
from("ftp://user#hostname/directoryname")
.process(new Processor() {
public void process(Exchange exchange) throws Exception
{
//my fantastic prosessing goes here
}
}).to("jms:queue:queueName");
the route could be way more complex than this, but it doesn't matter here. Imagine your solution is such a spectacular success that there's a plan to implement a service where anyone could add his ftp server and get the files processed and uploaded to the queue. So what you want is
(Flexibility) Being able to dynamically add/remove servers from your app
(Scaling) Being able to handle potentially large number of such servers
Let's forget about #2 and focus on Flexibility part.
So the question is, I suppose:
How to dynamically (at runtime) add/remove endpoints to/from Apache Camel route?
What I considered so far:
First, I admit I am not that familiar with Integration Patterns, but just scanning the catalogue, the only thing that kind of could fit the bill is the Content Enricher. It can take a message and just go off to somewhere else and bring sth else. So I was thinking if someone adds an ftp server, the connection details could be encapsulated in the message and a Content Enricher could then connect to that ftp server and fetch files and push it further through the route.... so it would effectively be a Content Enricher capable of connecting to multiple ftp servers.... That kind of sound wrong. First I don't think this is the intention behind that pattern and secondly since there's ftp Component in Camel, I should somehow be able to use it in that scenario
The second approach would be to break the route into two like using the vm component, like this:
from("ftp://user#hostname/directoryname").to("vm:internalQ");
from("vm:internalQ")
.process(new Processor() {
public void process(Exchange exchange) throws Exception
{
//my fantastic prosessing goes here
}
}).to("jms:queue:queueName");
so now, I can create many routes with ftp endpoints that write to that internal queue so it can be picked up. Adding route dynamically to CamelContext seems to be possible (Add camel route at runtime in Java). Is that the way to go? Or am I just trying to use Camel in a way that it was not designed to?
You can dynamically add routes to your CamelContext:
MyRouteBuilder trb = new MyRouteBuilder(servletEndpoint, mockEndpoint);
camelContext.addRoutes(trb);
And MyRouteBuilder:
MyRouteBuilder(Endpoint servletEndpointStart, MockEndpoint mockEndpointEnd, String allowedParameters){
this._servletEndpoint = servletEndpointStart;
this._mockEndpoint = mockEndpointEnd;
}
#Override
public void configure() throws Exception {
from(this._servletEndpoint)
.id(TESTING_ROUTE_NAME)
.process(new Processor(){ // some processor })
.to(_mockEndpoint);
}
You can also modify the route, but you will need to restart it, in order to work properly, checkout how it is done in:
org.apache.camel.model.RouteDefinition.adviceWith(ModelCamelContext, RouteBuilder)
I have a route in camel which is as follows
errorHandler(deadLetterChannel("file:somelocation");
from("jms:queuwlocation").to(
"file:someLocation");
I have read that camel error handling comes when there's a processing between two nodes, such as Processor, Predicate etc. But what if I cannot consume the message, in my case the camel cannot connect to jms. How should I log this exception ?. As I am trying to use a deadLetterChannel to send the message to filesystem, but since I have not received the message there is nothing new on the file location. how should I encounter this type of situation?
The problem you are facing might be due to the JMS Connection Exception being thrown outside of the life cycle of your Camel exchange. Claus Ibsen refers this in "Camel in Action" as a chicken and egg situation:
You can picture this as a chicken and egg situation. Camel's error handler only applies
during routing of Exchanges (chicken), but the consumer needs successfully to create the
Exchange (hatch the egg). So if we want a chicken but only have an egg, what can we do?
The answer lies with extending the error handling boundaries to cover the entire Camel JMS Consumer:
As
a figure of speech we can tell Camel to treat the eggs as if they were chickens. This is done by
configuring the consumer to bridge its internal error handler with Camel's error handler.
For the JMS module, I suspect this involves playing around with the transferException property. Good luck with that!
Futhermore, I don't think using a deadletter channel is appropriate for your problem, since you don't gain anything from taking messages out of the queue when encountering a connectivity problem.
Typically connectivity problems are self fixing, e.g. a server was restarted, and can be solved by retrial. For your use case, a sensible redelivery strategy can do this. Luckily Camel is really good at this: https://camel.apache.org/redeliverypolicy.html.
I recommend an indefinite redelivery until connectivity commences. Something like this:
onException(SomeJmsTimeoutException.class, SomeJmsConnectivityException.class)
.useOriginalMessage()
.maximumRedeliveries(Integer.MAX_VALUE)
.retryAttemptedLogLevel(LoggingLevel.ERROR)
.logRetryStackTrace(true)
.redeliveryDelay(1000 * 60);
As far as I know the error handling does not come in play just between the nodes. It has a more complete scope than that. It also depends on the type of error you are facing.
How you should log the exception?
Well why don't you do it like this?
DeadLetterChannel defined and you refer to it on your CamelContext.
The deadletteruri refers to another route - call it errorhandler route.
In the errorhandler route you can set headers that contain errors and other information you want to set.
In the errorhandler route in your "to" send the error message to a file writing the headers with it like you would with a log file.
I have an application I coded which I am refactoring to make better use of Spring Integration. The application processes the contents of files.
The problem (as I see it) is that my current implementation passes Files instead of Messages, i.e. Spring Integration Messages.
In order to avoid further rolling my own code, which I then have to maintain later, I'm wondering if there is a recommended structure for constructing Messages in Spring Integration. What I wonder is if there is some recommended combination of channel with something like MessageBuilder that I should use.
Process/Code (eventually)
I don't yet have the code to configure it but I would like to end up with the following components/processes:
Receive a file, remove header and footer of the file, take each line and convert it into a Message<String> (This it seems will actually be a Splitter) which I send on to...
Channel/Endpoint sends message to Router
Router detects format String in Payload and routes to the appropriate channel similar to Order Router here...
Selected channel then builds appropriate type of Message, specifically typed messages. For example I have the following builder to build a Message...
public class ShippedBoxMessageBuilder implements CustomMessageBuilder {
#Override
public Message buildMessage(String input) {
ShippedBox shippedBox = (ShippedBox) ShippedBoxFactory.manufactureShippedFile(input);
return MessageBuilder.withPayload(shippedBox).build();
}
...
Message is routed by type to the appropriate processing channel
My intended solution does seem like I've complicated it. However, I've purposefully separated two tasks 1) Breaking a file into many lines of Messages<String> and 2) Converting Messages<String> into Messages<someType>. Because of that I think I need an additional router/Message builder for the second task.
Actually, there is MessageBuilder support in the Spring Integration.
The general purpose of such Frameworks is to help back-end developers to decouple their domain code from messaging infrastructure. Finally, to work with Spring Integration you need to follow the POJO and Method Invocation principles.
You write your own services, transformers and domain models. Then you just use some out of the box compoenents (e.g. <int-file:inbound-channel-adapter>) and just refer from there to your POJOs, but not vise versa.
I recommend you to read Spring Integration in Action book to have more pictures on the matter.
Can you explain the reason to get deal with Spring Integration components directly?
UPDATE
1) Breaking a file into many lines of Messages
The <splitter> is for you. You should write some POJO which returns List<String> - the lines from your file without header and footer. How to read lines from File isn't a task of Spring Integration. Especially, if the "line" is something logical, not the real file line.
2) Converting Messages into Messages
One more time: there is no reason to build Message object. It's just enough to build new payload in some transformer (again POJO) and framework wrap to its Message to send.
Payload Type Router speaks for itself: it checks a payload type, but not Message type.
Of course, payload can be Message too, and even any header can be as well.
Anyway your Builder snapshot shows exactly a creation of plain Spring Integration Message in the end. And as I said: it will be enough just to transform one payload to another and return it from some POJO, which you will use as a transformer reference.
Please excuse stupidity as this is my first Camel application
To respond to a web request, I am sourcing the content from two different sources.
I am, therefore, making a multicast request to two methods and parallelizing it.
The response is an marshalled JSON object (using camel-jackson)
All works fine.
public class RestToBeanRouter extends RouteBuilder{
#Override
public void configure() throws Exception {
from("cxfrs://bean://rsServer")
.multicast()
.parallelProcessing()
.aggregationStrategy(new CoreSearchResponseAggregator())
.beanRef("searchRestServiceImpl", "firstMethod")
.beanRef("searchRestServiceImpl", "secondMethod")
.end()
.marshal().json(JsonLibrary.Jackson)
.to("log://camelLogger?level=DEBUG");
}
Question :
The Multicast routing expects a to in the DSL. Currently, I am mapping this to a log endpoint. Is this fine?
Since I am not using the to and the last exchange of the Aggregator strategy is the one which is returned to the user, should my endpoint be configured to something else - like a null or something? (Ah, the stupidity kicks in)
For the benefit of SO visitors, copying the solution given in the Camel mailing list here :
by Robert Simmons Jr. MSc. - Lead Java Architect # EA
Author of: Hardcore Java (2003) and Maintainable Java (2012)
The aggregated exchange is the one that gets returned and how the
aggregated exchange is created depends on the aggregation strategy you use.
When a route stops either by calling stop or merely not routing anymore,
the exchange on the last part of the route could be considered a reply. In
most cases it will reply back to the caller (unless you set a reply-to
destination in a JMS based route or some other cases). In your case if all
you want to do is return the enriched exchange then you dont need any to()
call. Just stop after the marshal.