Here is a simplified ftp polling mechanism.
<camelContext id="Fetcher" xmlns="http://camel.apache.org/schema/blueprint">
<redeliveryPolicyProfile id="redeliveryPolicy"
redeliveryDelay="10000"
maximumRedeliveries="-1" />
<camel:route id="fetchFiles">
<camel:from uri="ftp://10.20.30.40/From?username=user&password=RAW({{password}})&delay=3000" />
<camel:to uri="log:input?showAll=true&level=INFO"/>
<camel:to uri="file://incomingDirectory" />
<onException redeliveryPolicyRef="msRedeliveryPolicy">
<exception>java.lang.Exception</exception>
<redeliveryPolicy logRetryAttempted="true" retryAttemptedLogLevel="WARN"/>
</onException>
</camel:route>
</camelContext>
What do you think happens on failure? (Delay is 3 seconds, and
redeliveryDelay is 10 seconds.)
Answer: It polls every 3 seconds, forever.
So let's look at the docs. Maybe I need this
"repeatCount (scheduler)"
Specifies a maximum limit of number of fires. So if you set it to 1, the scheduler will only fire once. If you set it to 5, it will only fire five times. A value of zero or negative means fire forever.
Default: 0
Nope, it's not even a valid parameter. So why's it in the docs?
Unknown parameters=[{repeatCount=5}]
Ok, so I suppose every 3 seconds it polls. So how do I tell camel to stop that? Let's try set 'handled' to true?
<onException redeliveryPolicyRef="msRedeliveryPolicy">
<exception>java.lang.Exception</exception>
<redeliveryPolicy logRetryAttempted="true" retryAttemptedLogLevel="WARN"/>
<handled><constant>true</constant></handled>
</onException>
No luck. Still 3 seconds. It's clearly not even getting to the redelivery part.
What's the secret?
The fact is errors happen in from endpoint are not handled by user defined route (i.e. fetchFiles in above setup). So, onException and redeliveryPolicy are not involved as they only affect stuff belongs to user defined route.
To control the behavior of consumer defined in from endpoint, the obvious way is to use the option exist in that component. As suggested by #Screwtape, use backoffErrorThreshold and backoffMultplier for your case.
Why parameter repeatCount exist in doc, but is invalid to use? It probably does not exist in your camel version and Camel document writer forget to mark the first exist version in the doc.
Related
I have a problem with ActiveMQ similar to this one:
http://activemq.2283324.n4.nabble.com/Messages-stuck-in-pending-td4617979.html
and already tried the solution posted here.
Some messages seem to get stuck on the queue and can sit there for literally days without being consumed. I have more than enough consumers that are free most of the time, so it's not an issue of "saturation" of consumers.
Upon restart of the ActiveMQ SOME of the pending messages are consumed right away. Just a moment ago I had situation where I had 25 free consumers avaiable for queue (they are visible in the admin panel) with 7 of those "stuck" messages. Four of them were consumed right away but other 3 are still stuck. The other strange thing is - new messages kept coming to queue and were consumed right away, while the 3 old ones were still stuck.
On the consumer side my config in spring looks as follows:
<jms:listener-container concurrency="${activemq.concurrent.consumers}" prefetch="1">
<jms:listener destination="queue.request" response-destination="queue.response" ref="requestConsumer" method="onRequest"/>
</jms:listener-container>
<bean id="prefetchPolicy" class="org.apache.activemq.ActiveMQPrefetchPolicy">
<property name="queuePrefetch" value="1" />
</bean>
<bean id="connectionFactory" class="org.apache.activemq.spring.ActiveMQConnectionFactory">
<property name="brokerURL" value="${activemq.broker.url}?initialReconnectDelay=100&maxReconnectDelay=10000&startupMaxReconnectAttempts=3"/>
<property name="prefetchPolicy" ref="prefetchPolicy"/>
</bean>
The "stuck" messages are probably considered as "in delivery", restarting the broker will close the connections and, as the message are yet not acknowledged, the broker considers them as not delivered and will deliver them again.
There may be several problem leading to such a situation, most common ones are a problem in transaction / acknowledgment configuration, bad error / acknowledgment management on consumer side (the message is consumed but never acknowledged) or consumer being stuck on an endless operation (for example a blocking call to a third party resource which doesn't respond and there is no timeout handling).
I have a spring integration configuration file like:
<int-jms:inbound-channel-adapter
channel="fromjmsRecon"
jms-template="jmsTemplate"
destination-name="com.mycompany.inbound.recon">
<int:poller fixed-delay="3000" max-messages-per-poll="1"/>
</int-jms:inbound-channel-adapter>
<int:publish-subscribe-channel id="fromjmsRecon"/>
<int:service-activator input-channel="fromjmsRecon"
ref="processInboundReconFile"
method="execute"/>
... 10 More inbound channels ...
<int-jms:inbound-channel-adapter
channel="fromjmsVanRecon"
jms-template="jmsTemplate"
destination-name="com.mycompany.inbound.another">
<int:poller fixed-delay="3000" max-messages-per-poll="1"/>
</int-jms:inbound-channel-adapter>
<int:publish-subscribe-channel id="fromjmsVanRecon"/>
<int:service-activator input-channel="fromjmsVanRecon"
ref="processInboundAnother"
method="execute"/>
</beans>
There are 11 inbound-channel-adapter's. The first 10 connect to ActiveMQ, but the 11th one never does. It does not matter the order in which these adapters are listed, the 11th one always is ignored. The service adapter is initialized, but the channel adapter never connects to ActiveMQ.
Is there a limit to the number of inbound channel adapters? Is there a property that I can set somewhere that changes this limit?
Thanks for your help.
Correct, there is limit called TaskScheduler thread pool with size 10:
http://docs.spring.io/spring-integration/reference/html/configuration.html#namespace-taskscheduler
So, consider to change its size with spring.integration.taskScheduler.poolSize property, of use TaskExecutor for those adapters to shift tasks to other threads and don't eat expensive TaskScheduler.
There is other approach: don't use <int-jms:inbound-channel-adapter>, but switch to <int-jms:message-driven-channel-adapter>, which is listening by nature and much better.
How can I shutdown a Camel route context disgracefully?
As soon as I click the button, the Camel route should stop immediately. I don't want any delay.
Each time I do a camelroute.context.stop(), it takes some time to stop, and in that time since the route was active earlier queues and dequeues the messages are sent to the target queue.
I want to stop the route mid-way when I click the desired button.
Is there a way to handle it?
Have a look at the timeout property of the DefaultShutdownStrategy.
Try setting it to zero in your Camel Context:
<bean id="shutdownStrategy" class="org.apache.camel.impl.DefaultShutdownStrategy">
<property name="timeout" value="0"/>
</bean>
The value is in seconds by default.
Also, have a look at Graceful Shutdown in the Camel docs, if you haven't yet.
EDIT 1: The DefaultShutdownStrategy does not allow 0 timeouts. You could try setting it to 1 NANOSECOND which might help:
<bean id="shutdownStrategy" class="org.apache.camel.impl.DefaultShutdownStrategy">
<property name="timeout" value="1"/>
<property name="timeUnit" value="NANOSECONDS" /
</bean>
Alternatively, you can implement your own ShutdownStrategy if it's really important for you to guarantee absolute immediate shutdown.
I have the following route...
<route id="VM01_spit_products">
<from uri="direct:processXML" />
<split>
<method ref="CamelSplitOnKey" method="splitIntoBatches" />
<to uri="vm:processXMLSplit" />
</split>
</route>
<route id="VM01_processXML">
<from uri="vm:processXMLSplit?concurrentConsumers=15" />
<bean ref="Builder" method="createXMLFile" />
<to uri="{{ChangeReceiver}}" />
</route>
I was expecting with the use of VM or SEDA to mean that if the spliter is producing 5 messages, then one of the 15 threads I have defined would pick up each of these messages. When I debug into the Builder class, I can see that the messages are being picked up sequentially.
I see the same if I an using VM or SEDA.
Can someone suggest where I'm going wrong?
Notes:
Camel 2.6 due to JDK 1.5
New info.
I've added this code into my Builder.java
SedaEndpoint seda = (SedaEndpoint) camelContext.getEndpoint("seda:processXMLSplit");
int size = seda.getExchanges().size();
System.out.println("size ["+size+"]");
This prints a size of 0 each time.
This makes me think that the split isn't queuing up the messages as I expect.
Even if you have defined your vm consumer to have 15 threads it doesn't affect how your Split is working. By default Split is working sequentially so you must configure your Split to use parallelProcessing in order to get the result you want. See further in Splitter ParallelProcessing. Note also #Itsallas comment that you might need your vm endpoint configured with the same parameters.
I'm using Camel (2.11.0) to try and achieve the following functionality:
If a file exists at a certain location, copy it to another location and then begin processing it
If no such file exists, then I don't want the file consumer/poller to block; I just want processing to continue to a direct:cleanup route
I only want the file to be polled once!
Here's what I have so far (using Spring XML):
<camelContext id="my-camel-context" xmlns="http://camel.apache.org/schema/spring">
<route id="my-route
<from uri="file:///home/myUser/myApp/fizz?include=buzz_.*txt"/>
<choice>
<when>
<!-- If the body is empty/NULL, then there was no file. Send to cleanup route. -->
<simple>${body} == null</simple>
<to uri="direct:cleanup" />
</when>
<otherwise>
<!-- Otherwise we have a file. Copy it to the parent directory, and then continue processing. -->
<to uri="file:///home/myUser/myApp" />
</otherwise>
</choice>
<!-- We should only get here if a file existed and we've already copied it to the parent directory. -->
<to uri="bean:shouldOnlyGetHereIfFileExists?method=doSomething" />
</route>
<!--
Other routes defined down here, including one with a "direct:cleanup" endpoint.
-->
</camelContext>
With the above configuration, if there is no file at /home/myUser/myApp/fizz, then Camel just waits/blocks until there is one. Instead, I want it to just give up and move on to direct:cleanup.
And if there is a file, I see it getting processed inside the shouldOnlyGetHereIfFileExists bean, but I do not see it getting copied to /home/myUser/myApp; so it's almost as if the <otherwise> element is being skipped/ignored altogether!
Any ideas? Thanks in advance!
Try this setting, and tune your polling interval to suit:
From Camel File Component docs:
sendEmptyMessageWhenIdle
default =false
Camel 2.9: If the polling consumer did not poll any files, you can enable this option to send an empty message (no body) instead.
Regarding writing the file, add a log statement inside the <otherwise> to ensure it's being executed. If so, check file / folder permissions, etc.
Good luck.
One error i faced while I tried using the condition:
<simple>${body} != null</simple>
was it always returns true.
Please go through the below link:
http://camel.465427.n5.nabble.com/choice-when-check-BodyType-null-Body-null-td4259599.html
It may help you.
This is very old, but if anyone finds this, you can poll only once with
"?repeatCount=1"
I know the question was done almost 4 years ago but I had exaclty the same problem yesterday.
So I will let my answer here, maybe it will help another person.
I am using Camel, version 3.10.0
To make it work exactly as described in the question:
If a file exists at a certain location, copy it to another location and then begin processing it
If no such file exists, then I don't want the file consumer/poller to block; I just want processing to continue to a direct:cleanup route
ONLY want the file to be polled once!
Using ${body} == null
The configuration that we need are:
sendEmptyMessageWhenIdle=true //Will send a empty body when Idle
maxMessagesPerPoll=1 //Max files that it will take at once
repeatCount=1 //How many times will execute the Pool (above)
greedy=true // If the last pool executed with files, it will
execute one more time
XML:
<camel:endpoint id="" uri="file:DIRECTORY?sendEmptyMessageWhenIdle=true&initialDelay=100&maxMessagesPerPoll=1&repeatCount=1&greedy=false" />