I created a camel route that reads an xml an puts a message in an ActiveMQ-Queue for every element in xml. All is working fine but the error-handling does not work. Here is the route:
#Override
public void configure() throws Exception {
from(fileIn)
.routeId(IN_ROUTE_ID)
.onCompletion()
.to(CommonRoutes.ENDPOINT_DIRECT_LOGGING)
.end()
.onException(UncategorizedJmsException.class)
.maximumRedeliveries(maxRetries)
.backOffMultiplier(multiplier)
.redeliveryDelay(initialDelay)
.end()
.unmarshal(jaxbDataFormat)
.process(fileProcessor)
.split(body()).stopOnException().shareUnitOfWork()
.process(domainWrapProcessor)
.to(ACTIVE_MQ_OUT + ":queue:" + checkNotNull(queueName))
.end()
;
}
And here the configuration of FileEndpoint:
public void init(){
this.setAutoCreate(false);
this.setFile(new File(checkNotNull(csvFolder)));
this.setCamelContext(checkNotNull(context));
this.setAntInclude(ANT_INCLUDE);
this.setMove(doneFolder);
this.setMoveFailed(errorFolder);
}
The intention is that on every exception the file is moved to moveFailed folder and in case of JMSException (ActiveMQ offline) camel should retry and if finally fails also move to moveFailed.
I have created a unit tests for both cases. here is the test for any exception:
#Test
public void testException throws InterruptedException {
amqMmock.setExpectedMessageCount(3);
amqMock.whenAnyExchangeReceived((e) -> {
throw new RuntimeException("");
});
assertMockEndpointsSatisfied();
Thread.sleep(1000);
errorContainsFile(true);
}
This test is passing the file is moved into error folder set. But this is not working if i run this route in real (camel blueprint on fuse):
The problem is that on any (not UncategorizedJmsException) exception the file just keeps locked (.camelLock file created). (thanks to stopOnException the desired behavior for JMSExceptions is working)
So how to get camel to move the file to moveFailed folder on any exception?
To the best of my knowledge, moveFailed will only actually move the file to the error-dir if something goes wrong within the file-component itself (at least what I've experienced). This would explain why your file is not moved to the error-dir upon hitting an exception.
What you could try to do is perhaps something like this:
.onException(UncategorizedJmsException.class)
.maximumRedeliveries(maxRetries)
.backOffMultiplier(multiplier)
.redeliveryDelay(initialDelay)
.to(errorFolder)
.end()
Here is how I would have set up your route using blueprint with XML rather than Java DSL (well, roughly, just to get the gist of it):
<!-- Redelivery Policy -->
<bean id="redeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
<property name="maximumRedeliveries" value="${redelivery.max.attempts}" />
<property name="redeliveryDelay" value="${redelivery.delay.ms}" />
<property name="logRetryAttempted" value="${redelivery.log.attempts}" />
</bean>
<!-- Error Handler -->
<bean id="errorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
<property name="deadLetterUri" value="direct:error" />
<property name="useOriginalMessage" value="true" />
<property name="redeliveryPolicy" ref="redeliveryPolicyConfig" />
</bean>
<camelContext>
<route id="IN_ROUTE_ID">
<from uri="file://{{fileIn}}?move={{doneFolder}}&moveFailed={{errorFolder}}" />
...
<to uri="activemq:queue:{{queueName}}" />
</route>
<route id="error-route">
<from uri="direct:error" />
<to uri="file://{{errorFolder}}" />
</route>
</camelContext>
Related
I am using Spring Integration RecursiveDirectoryScanner to scan a directory recursively to process the incoming file that will be placed under the configured directory (/home/test).
I am frequently getting the below error:
ERROR org.springframework.integration.handler.LoggingHandler - java.lang.IllegalArgumentException: java.nio.file.FileSystemException: /home/test: Too many open files
at org.springframework.integration.file.RecursiveDirectoryScanner.listFiles(RecursiveDirectoryScanner.java:89)
at org.springframework.integration.file.FileReadingMessageSource.scanInputDirectory(FileReadingMessageSource.java:387)
at org.springframework.integration.file.FileReadingMessageSource.doReceive(FileReadingMessageSource.java:361)
at org.springframework.integration.file.FileReadingMessageSource.doReceive(FileReadingMessageSource.java:90)
at org.springframework.integration.endpoint.AbstractMessageSource.receive(AbstractMessageSource.java:134)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:224)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:245)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:58)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:190)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:186)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:353)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: /home/test: Too many open files
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
at java.nio.file.Files.newDirectoryStream(Files.java:457)
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300)
at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
at java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:72)
at java.nio.file.Files.walk(Files.java:3574)
at org.springframework.integration.file.RecursiveDirectoryScanner.listFiles(RecursiveDirectoryScanner.java:73)
My Spring Integration flow is as below:
Configuration in XML
<task:executor id="pollerPool"
pool-size="${pollerThreadPoolSize}"
queue-capacity="${pollerThreadQueueCapacity}" rejection-policy="ABORT" />
<task:executor id="fileHandlerPool"
pool-size="${fileHandlerPoolSize}"
queue-capacity="${fileHandlerPoolThreadQueueCapacity}" rejection-policy="CALLER_RUNS" />
<bean id="iFilter" class="org.springframework.integration.file.filters.ChainFileListFilter">
<constructor-arg>
<list>
<bean id="lastModifiedFileListFilter" class="org.springframework.integration.file.filters.LastModifiedFileListFilter">
<property name="age" value="120" />
</bean>
<ref bean="acceptOnceFileListFilter"/>
<bean class="org.springframework.integration.file.filters.RegexPatternFileListFilter">
<constructor-arg value="^.*\.(txt|csv|xls|xlsx|asc)$"/>
</bean>
</list>
</constructor-arg>
</bean>
<bean id="acceptOnceFileListFilter" name="acceptOnceFileListFilter" class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" primary="true" />
<bean id="recursiveDirectoryScanner" class="org.springframework.integration.file.RecursiveDirectoryScanner">
<property name="filter" ref="iFilter" />
<property name="locker" ref="nioFileLocker" />
</bean>
<bean id="nioFileLocker" class="org.springframework.integration.file.locking.NioFileLocker" />
<int-file:inbound-channel-adapter
id="fileSource" channel="fileReceivedChannel" auto-startup="true"
directory="file:${polling.directory}"
scanner="recursiveDirectoryScanner" >
<int:poller task-executor="pollerPool"
fixed-rate="${pollerFixedRate}"
receive-timeout="${pollerReceiveTimeout}">
</int:poller>
</int-file:inbound-channel-adapter>
Dynamic parameters are as below:
polling.directory=/home/test
pollerThreadPoolSize=1
pollerThreadQueueCapacity=10
pollerFixedRate=5000
pollerReceiveTimeout=5000
fileHandlerPoolSize=2
fileHandlerPoolThreadQueueCapacity=100
EDIT:
I do unlock file in a service activator that comes in to picture when a file is picked. I get some information from file and unlock it.
#Autowired
NioFileLocker nioFileLocker;
protected void doTransform(Message<?> message) throws Exception {
MessageBuilder<File> payload = (MessageBuilder<File>) message.getPayload();
File inFile = payload.getPayload();
try {
nioFileLocker.unlock(inFile);
} catch (Exception e) {
LOGGER.error("file not unlock");
}
}
Is there any issue with the configuration ? How do I make sure this exception never appear again ?
Thank you in advance.
I would suggest to test your solution without NioFileLocker. Doesn't look like you are using it for unlocking files, but the lock(File fileToLock) really keeps some file marker in OS.
On the other hand the file locker doesn't work reliable on UNIX systems. It still allows an access to files. At least for reading.
For better exclusive file access I would recommend to use a FileSystemPersistentAcceptOnceFileListFilter with the external MetadataStore instead of in memory AcceptOnceFileListFilter. This way only one instance of your application will get access to the file and it won't be processed again at all.
I'm trying to convert my route made with Java DSL in a route made with XML.
The following is my original route that works. And what it does is simple.
Get as input an array of integer and at runtime throws some errors.
At the end of the route I need to read all the error raised by myself and not with a long stacktrace or other long messages in console.
other lines of code...
camelContext.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
onException(Exception.class)
.handled(true)
.process(new ErrorProcessor());
from("direct:start_route")
.split(body())
.processRef("payloadProcessor")
.to("direct:a")
.end()
.to("direct:d");
from("direct:a")
.beanRef("stupidBean", "stupidMethod");
from("direct:d")
.beanRef("errorProcessor", "check");
}
});
other lines of code...
The following is my xml route that doesn't work.
...
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="direct:call.playWithPayload" />
<onException>
<exception>java.lang.Exception</exception>
<process ref="errorProcessor" />
</onException>
<split onPrepareRef="payloadProcessor">
<simple>${body}</simple>
<to uri="direct:call.a" />
</split>
<to uri="direct:call.d" />
</route>
<!--SUB ROUTE-->
<route>
<from uri="direct:call.a" />
<bean ref="stupidBean" method="stupidMethod" />
</route>
<route>
<from uri="direct:call.d" />
<bean ref="errorProcessor" method="check" />
</route>
</camelContext>
...
What I need is that the direct:call.d is called after the split.
Thanks to this I can read all the errors added into a List<Exception> that is stored into the header.
I think that the problem is in the onException management.
When I try to add the handled to reproduce the my Java DSL
<onException>
...
<handled>
<constant>
true
</constant>
</handled>
...
I got this error:
Invalid content was found starting with element 'handled'.
One of '{"http://camel.apache.org/schema/spring":description, "http://camel.apache.org/schema/spring":exception}' is expected.
Found solution.
My problem was an incorrect format of my xml route.
<onException>
<exception>java.lang.Exception</exception>
<handled>
<constant>true</constant>
</handled>
<process ref="errorProcessor" />
</onException>
now it works.
I am new to apache camel. I want to migrate from service mix file poller to camel file poller. I am trying to do it, but currently I have nothing to test as I have to code this and give someone for testing. So can someone help me and check whether I am going in right way?
Service-mix File Poller code:
<sm:activationSpec componentName="abcFilePoller"
destinationService="b:destinationA"
service="b:abcFilePoller">
<sm:component>
<bean class="org.apache.servicemix.components.file.FilePoller">
<property name="file" value="file://D:/input" />
<property name="period" value="20000"/>
<property name="archive" value="file://D:/archive" />
<property name="filter" ref="abcFileFilter" />
<property name="marshaler">
<bean class="org.apache.servicemix.components.util.BinaryFileMarshaler" />
</property>
</bean>
</sm:component>
</sm:activationSpec>
<sm:activationSpec componentName="destinationA"
service="b:destinationA">
<sm:component>
<bean
class="com.abc.file.ABCReceiverComponent">
</bean>
</sm:component>
</sm:activationSpec>
<bean id="abcFileFilter" class="org.apache.commons.io.filefilter.WildcardFileFilter">
<constructor-arg value="A*.ID" />
Apache Camel File Poller
<camel:route id="abcFilePoller">
<camel:from
uri="timer://time?period=20000"/>
<camel:pollEnrich uri="file://D:/input"/>
<camel:filter ref="abcFileFilter"></camel:filter>
<camel:to uri="file://D:/archive" />
<camel:to uri="" />
</camel:route>
<bean id="abcFileFilter" class="org.apache.commons.io.filefilter.WildcardFileFilter">
<constructor-arg value="A*.ID" />
</bean>
I have not completed the camel coding. I have left with destination part. And I have no idea about the marshaler that is used in the service-mix part. How to implement that BinaryFileMarshaler using camel.
You can do this even easier in Apache Camel where you can configure the filtering in the file endpoint, so it just becomes
<route>
<from uri="file:D:/input?delay=20000&include=A.*ID"/>
<to uri="file:D:/archive"/>
</route>
Just mind that the include option uses a regular expression, so if you are not familiar with that it can take a bit tries to get the expression to work as expected. But its standard java regular expressions.
See more at: https://camel.apache.org/components/latest/file-component.html
And for new users to Apache Camel then see: http://java.dzone.com/articles/open-source-integration-apache
I'm new to ActiveMQ (version 5.9.1) and Apache Camel. I'm experimenting about a couple of routes. I wrote them in Java, then packed in a .jar and deployed on %ACTIVEMQ_HOME%/lib. In my custom foo-activemq.xml I added a camelContext tag with a package child tag. Here is my relevant .xml
...
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<package>edu.foo.amq.camel</package>
</camelContext>
...
And here are my two routes:
package edu.foo.amq.camel;
import org.apache.camel.builder.RouteBuilder;
public class NumberRoute extends RouteBuilder {
#Override
public void configure() throws Exception {
from("jms:queue:number.queue")
.marshal("UTF-8")
.choice()
.when(header("readyToGo").isNotEqualTo(true))
.to("jms:queue:big.number.queue");
}
}
and
package edu.foo.amq.camel;
import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
public class BigNumberRoute extends RouteBuilder {
#Override
public void configure() throws Exception {
from("jms:queue:big.number.queue")
.marshal("UTF-8")
.split(body().tokenize("\n")).streaming()
.process(new Processor() {
#Override
public void process(Exchange arg0) throws Exception {
arg0.getIn().setHeader("readyToGo", true);
System.out.println(arg0.getIn().getBody(
String.class));
}
})
.to("jms:queue:number.queue");
}
}
When I start my ActiveMQ broker I can see ... Total 0 routes, of wich 0 is started.. If I look at the ActiveMQ web console I can se the queues number.queue and big.number.queue up and running but my java routes don't. What am I missing?
UPDATE:
I remove the camelContext tag from my foo-activemq.xml configuration file and add the import tag instead:
...
<import resource="jetty.xml"/>
<import resource="foo-camel.xml"/>
...
Here is my foo-camle.xml configuration file, I largely use the %ACTIVEMQ_HOME%/examples/conf/camel.xml file. I put this file into %ACTIVEMQ_HOME%/conf:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://camel.apache.org/schema/spring
http://camel.apache.org/schema/spring/camel-spring.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<package>edu.foo.amq.camel</package>
</camelContext>
<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" >
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="vm://amq-broker?create=false"/>
<property name="userName" value="${activemq.username}"/>
<property name="password" value="${activemq.password}"/>
</bean>
</property>
</bean>
but the result is the same: ActiveMQ says no routes
The Camel configuration XML file must be added to the activemq.xml configuration file:
<import resource="foo-activemq.xml" />
EDIT:
What else?
Rename foo-activemq.xml to the default activemq.xml
Verify again if your jar is really in the lib directory (I am sure you have done that already)
Add a simple test route to foo-camel-xml and check in the log file if this route is loaded, e.g.
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring" >
<package>edu.foo.amq.camel</package>
<route>
<from uri="direct:start" />
<log message="${body}" />
</route>
</camelContext>
I'm dealing with a problem how to read a file with Camel in SMX 4.4.2 but only in case when
the read operation cannot be performed in Camel route from.
I need to read data from the file transform it into XML and then return it to the requestor (the requestor hits SMX via REST Service).
I'm having serious problems with reading the file after the REST invokation.
Routes are like these:
<jaxrs:server id="restService" address="http://localhost:9000/REST/"
staticSubresourceResolution="true">
<jaxrs:serviceBeans>
<ref bean="restFtpBean" />
</jaxrs:serviceBeans>
</jaxrs:server>
<bean id="restFtpBean" class="poc.rest.RESTFtpProxyService" />
<cxf:rsServer id="rsServer" address="http://localhost:9000/REST/"
serviceClass="poc.rest.RESTFtpProxyService" />
<camel:camelContext xmlns="http://camel.apache.org/schema/spring">
<endpoint id="ftpProvider" uri="ftp://localhost:21/?fileName=test.xml" />
<route>
<from uri="cxfrs://bean://rsServer" />
<to ref="ftpProvider" />
</route>
</camel:camelContext>
In this scenario Camel tries to write to file !
What am I doing wrong ?
Its always advised to learn and get familiar with the EIPs
http://camel.apache.org/content-enricher.html
Your use-case you need to consume a file from within a route, and there is an EIP pattern for that: http://camel.apache.org/content-enricher.html. See the section about using pollEnrich to consume/poll the file.