I have 3 routes:
route-file1 which reads file1.csv and converts to array
route-file2 which reads file2.csv and converts to array
route-final which uses enrich of both routes (with custom aggregator to merge arrays) and do something
Problem is if route-file1 succeeds, but route-file2 or any other routes fail. route-file1 already completed and moved file1.csv to .done folder, so I cannot rerun everything again.
Is there a way that when route-final fails, it propagates exception to other routes used in enrich? I tried using transaction which works fine with stopping the route execution, but does not propagate exception to the route-fileX routes. Is this possible with the camel?
You can set shareUnitOfWork to true in your content-enricher's so they work together in the same unit of work. See more details in the documentation: http://camel.apache.org/content-enricher.html
Related
I am working on an application that is using project-reactor Kafka APIs to connect reactively to Kafka-brokers. The use-case is that there is an input topic which contains file-paths to the files for processing. The application reads each file, processes it, creates a flux of the processed messages and pushes it to the output topic. The requirement is that the file must be deleted once it has been processed and processed messages should be pushed to the output topic. So, the delete action must be performed after each file has been processed and the flux of the message pushed to the output topic.
public Flux<?> flux() {
return KafkaReceiver
.create(receiverOptions(Collections.singleton(sourceTopic)))
.receive()
.flatMap(m -> transform(m.value()).map(x -> SenderRecord.create(x,
m.receiverOffset())))
.as(sender::send)
.doOnNext(m -> {
m.correlationMetadata().acknowledge();
deleteFile(path);
}).doOnCancel(() -> close());
}
*The transform() method initiates the file processing in the file path(m.value()) and returns a flux of messages.
The problem is that the file is deleted even before all the messages is pushed to output topic. Therefore, in case of a failure, on re-try the original file is not available.
Since it seems the path variable is accessible in the whole pipeline (method input parameter?), you could delete the file within a separate doFinally. You would need to filter for onComplete or cancel SignalType, because you don't want to delete the file in case of a failure.
Another option would be doOnComplete if you're not interested in deleting the file upon cancellation.
(question tldr at end)
So my task for the Mule "Transform Message" component is to take a bunch of user info from LDAP Directory Service and provide it to an old database endpoint using SOAP. Fairly simple transform stuff.
The main ! about this operation is the size of the message that has to be provided to the endpoint. The entire payload has to be provided in a single message, otherwise the service will remove all entries that are not part of the payload (there is no explicit 'delete' service). This is an issue because the amount of users in the directory is roughly 20,000 causing every message to be 5MB or so in size.
My flow in Mule Studio currently works with a low amount of users being returned from the LDAP component. Successful return from the endpoint and I can see the data updated in the legacy environment. When applying this to a more 'production-realistic' load the Web Service Consumer (SOAP) craps out with an odd exception (unexpected EOF/character).
So I stuck a File component in the middle to dumpcheck the message that was being sent to the Consumer. The message is indeed getting cut before it can finish, which is where the EOF is coming from.
This is the transform script in Dataweave.
%output application/xml
%namespace ns0 test.namespace.com
---
{
ns0#updateContact: {
ns0#ContactType: "Primary",
ns0#ContactDetails: {
(payload map {
(ns0#ContactDetailElem: {
ns0#personID: $.personID,
ns0#contactDetail: $.desc
}) when $.personID != null
})
}
}
}
The expected output is below and successfully occurs with a lesser payload.
<?xml version='1.0' encoding='windows-1252'?>
<ns0:updateContact xmlns:ns0="test.namespace.com">
<ns0:ContactType>Primary</ns0:ContactType>
<ns0:ContactDetails>
<../>
<ns0:ContactDetailElem>
<ns0:personID>{Integer}</ns0:personID>
<ns0:contactDetail>{String.detail}</ns0:contactDetail>
</ns0:ContactDetailElem>
<../>
</ns0:ContactDetails>
</ns0:updateContact>
On the big payload the following happens at the end of the file
<?xml version='1.0' encoding='windows-1252'?>
<ns0:updateContact xmlns:ns0="test.namespace.com">
<ns0:ContactType>Primary</ns0:ContactType>
<ns0:ContactDetails>
<../>
<ns0:ContactDetailElem>
<ns0:personID>{Integer}</ns0:personID>
<ns0:contactDetail>{String.detail}</ns0:contactDetail>
</ns0:ContactDeta
Which looks like a typo but is what looks like the message being cut before it can finish. The file size is always stopped at 3,553,099 characters. Of course this makes the flow crap out as the xml is invalid.
The question then is there a limit on the message size that the Dataweave transformer can create? If not a legitimate bug but a configuration issue, where would I find this setting? I've had a look around but can't find anybody encounter this type of issue.
TL;DR: Do Dataweave transform messages have a size limit around 3.38MB?
Exception caused by: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
PS: I've found the documentation on dataweave streaming after typing this up, will see if this can help my situation. Otherwise i'm considering implementing a workaround to construct the message outside dataweave and then passing it to the Consumer.
Are you using Mule version 3.8.3? Try 3.8.4, it fixed a bug in DataWeave which caused cutoff of Strings in some cases.
We have a similar problem, same as yours that is with the problem of size. We implemented streaming using stax.
I wanted to apply re-deliver and use Dead Letter Channel. So found this very useful apache-camel FAQ link. I followed the steps as mentioned in this website.
I added a little more logic, the code is available in github .
Basically, as per the FAQ, we have to split the routes, such that(as mentioned in the FAQ) the camel can handle any exception in the sub route.
Here is the code (Camel Route):
#Override
public void configure() throws Exception {
errorHandler(deadLetterChannel("mock:error")
.logExhaustedMessageBody(true)
.logExhausted(true)
.logRetryStackTrace(true)
.maximumRedeliveries(3)
.redeliveryDelay(1000)
.backOffMultiplier(2)
.useOriginalMessage()
.useExponentialBackOff());
from("direct:start")
.log("Dump incoming body: " + body())
.to("direct:sub")
.end();
from("direct:sub")
.errorHandler(noErrorHandler())
.process(new SubRouteProcessor())
.log("Dump incoming body: "+ body())
.process(new NewSubRouteProcessor())
.transform(body().append("Modified Data !"))
.to("mock:result");
}
I wrote a simple unit test, to trigger the exception, hence the redeliver code is executed.
Now the difficulty/problem: The expected behavior is to do only 3 redeliver attempts, with 1000 millis delay and ExponentialBackOff.
However, the Test runs forever, keeps on doing the re-delivery with exponential delay. If I remove/comment the 2 processor calls, the code does run fine, means on exception, it does only 3 retries.
Could you please help to understand,
1) what is wrong in this code/route?
2) what is causing code to run forever?
3) why removing these processors, it works,retry happens for
said number of times only?
I just want - when ever the "direct:sub" throws "exception", the control goes back to "direct:start", re-try processing message from a certain point.
Thank you !!
I checked out your code and simply by changing getOut to getIn in both of your processors, the test passes.
Edit: My suspicion from the beginning was that your processors were somehow overriding the headers Camel set in order to do a certain amount of redelivery. In your case:
CamelRedelivered=true, CamelRedeliveryCounter=1, CamelRedeliveryMaxCounter=3
The header CamelRedeliveryCounter was not incremented when you were setting the Body on the getOut exchange. In my mind, it should and this is a bug; but maybe it is working as designed? Maybe #claus-ibsen could help with that.
Hope this helps.
R.
I am trying to implement a process consisting of several webservice-calls, initiated by a JMS-message read by Spring-integration. Since there are no transactions across these WS-calls, I would like to keep track of how far my process has gone, so that steps that are already carried out are skipped when retrying message processing.
Example steps:
Retrieve A (get A.id)
Create new B for A (using A.id, getting B.id)
Create new C for B (using B.id, getting C.id)
Now, if the first attempt fails in step 3, I already have a created a B, and know it's id. So if I want to retry the message, it will skip the second step, and not leave me with an incomplete B.
So, to the question: Is it possible to decorate a JMS-message read by Spring-integration with additional header properties upon message processing failures? If so, how could I do this?
The way it works at the moment:
Message is read
Some exception is thrown
Message processing halts, and ActiveMQ places the message on DLQ
How I would like it to work:
Message is read
Some exception is thrown
The exception is handled, with the result of this handling being an extra header property added to the original message
ActiveMQ places the message on DLQ
One thing that might achieve this is the following:
Read the message
Start processing, wrapped in try-catch
On exception, get the extra information from the exception, create a new message based on the original one, add extra info to header and send it directly to the DLQ
Swallow the exception so the original message dissappears
This feels kinda hackish though, hopefully there is a more elegant solution.
It's hard to generalize without more information about your flow(s) but you could consider adding a custom request handler advice to decorate and/or re-route failed messages. See Adding Behavior to Endpoints.
As the other answer says, you can't modify the message but you can build a new one from it.
EDIT:
So, to the question: Is it possible to decorate a JMS-message read by Spring-integration with additional header properties upon message processing failures? If so, how could I do this?
Ahhh... now I think I know what you are asking; no, you can't "decorate" the existing message; you can republish it with additional headers instead of throwing an exception.
You can republish in the advice, or in the error flow.
It might seem like a "hack" to you, but the JMS API provides no mechanism to do what you want.
From the spring forum:
To place new header to the MessageHeaders you should use
MessageBuilder, because not only headers, but entire Message is
immutable.
return MessageBuilder.fromMessage(message).setHeader(updateflag, message.getHeaders().get("Lgg_Rid") == "ACK" ? "CONF" : "FAIL").build();
In an asynchronous context, errors will go to an error channel - either one you configure yourself and indicate in the message headers with errorChannel, or a global error channel if none is specified. See for more details here.
We are using Camel fluent builders to set up a series of complex routes, in which we are using dynamic routing using the RecipientList functionality.
We've encountered issues where in some cases, the recipient list contains a messaging endpoint that doesn't exist (for example, something like seda:notThere).
A simple example is something like this:
from("seda:SomeSource")....to("seda:notThere");
How can I configure the route so that if the exchange tries to route to an endpoint that doesn't already exist, an error is thrown?
I'm using Camel 2.9.x, and I've already experimented with the Dead Letter Channel and various Error Handler implementations, with (seemingly) no errors or warnings logged.
The only logging I see indicates that Camel is (attempting to) send to the endpoint which doesn't exist:
2013-07-03 16:07:08,030|main|DEBUG|o.a.c.p.SendProcessor|>>>> Endpoint[seda://notThere] Exchange[Message: x.y.Z#293b9fae]
Thanks in advance!
All endpoints behave differently in this case.
If you attempt to write to a ftp server that does not exist, you certainly get an error (connection refused or otherwise)..
This is also true for a number of endpoints.
SEDA queues gets created if the do not exist and the message will be left there. So your route actually sends to "notThere" and the message will still be there until the application restarts or someone starts to consume messages from seda:notThere. This is the way seda queues are designed. If you set the size of the seda queue by to("seda:notThere?size=100"), then if there is noone reading (or reading slowly) you will get exceptions on message 101 and forward.
If you need to be sure some route is consuming your messages, use "direct" instead of "seda". You can even have some middle layer to use the features of seda with respect to staging and the features of direct knowing there is a consumer active (if sent from recipient list with perhaps user input (god forbid).
from("whatever").recipentList( ... ); // "direct:ep1" work, "direct:ep2" throws exception
from("direct:ep1").to("seda:ep1");
from("seda:ep1").doRealStagedStuffHere();