I have a spring integration IntegrationFlow that's defined like this:
IntegrationFlows.from(Amqp.inboundAdapter(connectionFactory, "queueName")
.id("id")
.autoStartup(autoStartup)
.concurrentConsumers(2)
.maxConcurrentConsumers(3)
.messageConverter(messageConverter()))
.aggregate(a -> ...)
.handle(serviceActivatorBean)
.get();
And serviceActivatorBean looks like this:
#Component
#Transactional
public class ServiceActivator {
#ServiceActivator
public void myMethod(Collection<MyEvent> events) {
....
}
}
If myMethod throws an exception it will be logged but no retry will happen. I've tried to change the IntegrationFlow to this:
RequestHandlerRetryAdvice advice = new RequestHandlerRetryAdvice();
RetryTemplate retryTemplate = new RetryTemplate();
SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
retryPolicy.setMaxAttempts(5);
retryTemplate.setRetryPolicy(retryPolicy);
advice.setRetryTemplate(retryTemplate);
IntegrationFlows.from(Amqp.inboundAdapter(connectionFactory, "queueName")
.id("id")
.autoStartup(autoStartup)
.adviceChain(advice)
.concurrentConsumers(2)
.maxConcurrentConsumers(3)
.messageConverter(messageConverter()))
.aggregate(a -> ...)
.handle(serviceActivatorBean)
.get();
But then I a log message like this (an retries won't happen):
2017-06-30 13:18:10.611 WARN 88706 --- [erContainer#1-2]
o.s.i.h.a.RequestHandlerRetryAdvice : This advice
org.springframework.integration.handler.advice.RequestHandlerRetryAdvice
can only be used for MessageHandlers; an attempt to advise method
'invokeListener' in
'org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1'
is ignored
How can I configure this IntegrationFlow to behave the same way a RabbitListener would? I.e. let RabbitMQ publish the messages again.
Use a retry interceptor in the adapter's advice chain instead of the RequestHandlerRetryAdvice - that is for consuming endpoints, as the message says.
Related
I am new to the Spring Integration project, now I need to create a flow with Java DSL and test it.
I came up with these flows. First one should run by cron and invoke second one, which invokes HTTP endpoint and translates XML response to POJO:
#Bean
IntegrationFlow pollerFlow() {
return IntegrationFlows
.from(() -> new GenericMessage<>(""),
e -> e.poller(p -> p.cron(this.cron)))
.channel("pollingChannel")
.get();
}
#Bean
IntegrationFlow flow(HttpMessageHandlerSpec bulkEndpoint) {
return IntegrationFlows
.from("pollingChannel")
.enrichHeaders(authorizationHeaderEnricher(user, password))
.handle(bulkEndpoint)
.transform(xmlTransformer())
.channel("httpResponseChannel")
.get();
}
#Bean
HttpMessageHandlerSpec bulkEndpoint() {
return Http
.outboundGateway(uri)
.httpMethod(HttpMethod.POST)
.expectedResponseType(String.class)
.errorHandler(new DefaultResponseErrorHandler());
}
Now I want to test flow and mock HTTP call, but struggling to mock HTTP handler, I tried to do it like that:
#ExtendWith(SpringExtension.class)
#SpringIntegrationTest(noAutoStartup = {"pollerFlow"})
#ContextConfiguration(classes = FlowConfiguration.class)
public class FlowTests {
#Autowired
private MockIntegrationContext mockIntegrationContext;
#Autowired
public DirectChannel httpResponseChannel;
#Autowired
public DirectChannel pollingChannel;
#Test
void test() {
final MockMessageHandler mockHandler = MockIntegration.mockMessageHandler()
.handleNextAndReply(message -> new GenericMessage<>(xml, message.getHeaders()));
mockIntegrationContext.substituteMessageHandlerFor("bulkEndpoint", mockHandler);
httpResponseChannel.subscribe(message -> {
assertThat(message.getPayload(), is(notNullValue()));
assertThat(message.getPayload(), instanceOf(PartsSalesOpenRootElement.class));
});
pollingChannel.send(new GenericMessage<>(""));
}
}
But I am always getting an error, that on line:
mockIntegrationContext.substituteMessageHandlerFor("bulkEndpoint", mockHandler);
org.springframework.beans.factory.BeanNotOfRequiredTypeException: Bean named 'bulkEndpoint' is expected to be of type 'org.springframework.integration.endpoint.IntegrationConsumer' but was actually of type 'org.springframework.integration.http.outbound.HttpRequestExecutingMessageHandler'
Am I doing something wrong here? I am assuming I have a problem with IntegrationFlow itself, or maybe my testing approach is a problem.
The error is correct. The bulkEndpoint is not an endpoint by itself. It is really a MessageHandler. The endpoint is created from the .handle(bulkEndpoint).
See docs: https://docs.spring.io/spring-integration/docs/current/reference/html/overview.html#finding-class-names-for-java-and-dsl-configuration and https://docs.spring.io/spring-integration/docs/current/reference/html/testing.html#testing-mocks.
So, to make it working you need to do something like this:
.handle(bulkEndpoint, e -> e.id("actualEndpoint"))
And then in the test:
mockIntegrationContext.substituteMessageHandlerFor("actualEndpoint", mockHandler);
You also probably need to think to not have that pollerFlow to be started when you test it sine you send the message into pollingChannel manually. So, there is no conflicts with what you'd like to test. For this reason you also add a id() into your e.poller(p -> p.cron(this.cron)) and use #SpringIntegrationTest(noAutoStartup) to have it stopped before your test. I see you try noAutoStartup = {"pollerFlow"}, but this is not going to help for static flows. You indeed need to have stopped an actual endpoint in this case.
I am trying to configure a TCP socket that receives data in the format name,value in distinct messages. Those messages arrive on average every second, sometimes faster or sometimes slower.
I was able to set up a working configuration but I am lacking a basic understanding of what actually is happening in Spring Integration.
My configuration file looks like this:
#Configuration
#EnableIntegration
public class TCPSocketServerConfig
{
#Bean
public IntegrationFlow server(
final CSVProcessingService csvProcessingService,
#Value("${tcp.socket.server.port}") final int port
)
{
return IntegrationFlows.from(
Tcp.inboundAdapter(
Tcp.nioServer(port)
.deserializer(serializer())
.leaveOpen(true)
)
.autoStartup(true)
.outputChannel(queueChannel())
).transform(new ObjectToStringTransformer())
.handle(csvProcessingService)
.get();
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller()
{
return Pollers.fixedDelay(50, TimeUnit.MILLISECONDS).get();
}
#Bean
public MessageChannel queueChannel()
{
return MessageChannels.queue("queue", 50).get();
}
#Bean
public ByteArrayLfSerializer serializer()
{
final ByteArrayLfSerializer serializer = new ByteArrayLfSerializer();
serializer.setMaxMessageSize(10240);
return serializer;
}
}
And the CSVProcessingService looks like this (abbreviated):
#Slf4j
#Service
public class CSVProcessingService
{
#ServiceActivator
public void process(final String message)
{
log.debug("DATA RECEIVED: \n" + message);
final CsvMapper csvMapper = new CsvMapper();
final CsvSchema csvSchema = csvMapper.schemaFor(CSVParameter.class);
if (StringUtils.contains(message, StringUtils.LF))
{
processMultiLineInput(message, csvMapper, csvSchema);
}
else
{
processSingleLineInput(message, csvMapper, csvSchema);
}
}
}
My goals for this configuration are the following:
receive messages on the configured port
withstand a higher load without losing messages
deserialize the messages
put them into the queue channel
(ideally also log errors)
the queue channel is polled every 50 ms and the message from the queue channel passed to the ObjectToStringTransformer
after the transformer the converted message is passed to the CSVProcessingService for further processing
Did I achieve all those goals correctly or did I make a mistake because I misunderstood Spring Integration? Would it be possible to combine the Poller and the #ServiceActivator somehow?
Futhermore, I have a problem visualizing how my configured IntegrationFlow actually "flows", maybe somebody can help me to better understand this.
EDIT:
I reworked my configuration after Artems comment. It now look like this:
#Configuration
#EnableIntegration
public class TCPSocketServerConfig
{
#Value("${tcp.socket.server.port}") int port;
#Bean
public IntegrationFlow server(
final CSVProcessingService csvProcessingService
)
{
return IntegrationFlows.from(
Tcp.inboundAdapter(
tcpNioServer()
)
.autoStartup(true)
.errorChannel(errorChannel())
)
.transform(new ObjectToStringTransformer())
.handle(csvProcessingService)
.get();
}
#Bean
public AbstractServerConnectionFactory tcpNioServer()
{
return Tcp.nioServer(port)
.deserializer(serializer())
.leaveOpen(true)
.taskExecutor(
new ThreadPoolExecutor(0, 20,
30L, TimeUnit.SECONDS,
new SynchronousQueue<>(),
new DefaultThreadFactory("TCP-POOL"))
).get();
}
#Bean
public MessageChannel errorChannel()
{
return MessageChannels.direct("errors").get();
}
#Bean
public IntegrationFlow errorHandling()
{
return IntegrationFlows.from(errorChannel()).log(LoggingHandler.Level.DEBUG).get();
}
#Bean
public ByteArrayLfSerializer serializer()
{
final ByteArrayLfSerializer serializer = new ByteArrayLfSerializer();
serializer.setMaxMessageSize(10240);
return serializer;
}
}
I also removed the #ServiceActivator annotation form the CSVProcessingService#process method.
Not sure what confuses you, but your configuration and logic looks good.
You may miss the fact that you don't need a QueueChannel in between, since an AbstractConnectionFactory.processNioSelections() is already multi-threaded and it schedules a task to read a message from the socket. So, you only need is to configure an appropriate Executor for Tcp.nioServer(). Although it is an Executors.newCachedThreadPool() by default anyway.
On the other hand with in-memory QueueChannel you indeed may lose messages because they are already read from the network.
When you do Java DSL, you should consider to use poller() option on the endpoint. The #Poller will work on the #ServiceActivator if you have inputChannel attribute over there, but using the same in the handle() will override that inputChannel, so your #Poller won't be applied. Don't confuse yourself with mixing Java DSL and annotation configuration!
Everything else is good in your configuration.
I may just be missing something very simple here (or misusing something), but I was attempting to set up two direct channels such that one flow would pass some data to each sequentially. So using the Spring Integration JAVA DSL I had something like this (significantly simplified for this example):
public static final String TEST_CHANNEL = "testGateway";
public static final String TEST_UPPER_CHANNEL = "testChannelUpper";
public static final String TEST_LOWER_CHANNEL = "testChannelLower";
#Bean(name = TEST_CHANNEL)
public MessageChannel testGatewayChannel() {
return MessageChannels.direct(TEST_CHANNEL).get();
}
#Bean(name = TEST_UPPER_CHANNEL)
public MessageChannel testChannelUpperChannel() {
return MessageChannels.direct(TEST_UPPER_CHANNEL).get();
}
#Bean(name = TEST_LOWER_CHANNEL)
public MessageChannel testChannelLowerChannel() {
return MessageChannels.direct(TEST_LOWER_CHANNEL).get();
}
#Bean
public IntegrationFlow testFlow() {
return IntegrationFlows
.from(TEST_CHANNEL)
.channel(TEST_UPPER_CHANNEL)
.channel(TEST_LOWER_CHANNEL)
.get();
}
#Bean
public IntegrationFlow testUpperFlow() {
return IntegrationFlows
.from(TEST_UPPER_CHANNEL)
.<String, String>transform(String::toUpperCase)
.handle(System.out::println)
.get();
}
#Bean
public IntegrationFlow testLowerFlow() {
return IntegrationFlows
.from(TEST_LOWER_CHANNEL)
.<String, String>transform(String::toLowerCase)
.handle(System.out::println)
.get();
}
I'm using a REST endpoint to invoke the flow via a gateway, but when I do so it seems only one of the channels is invoked. The channel also seems to be random across invocations (sometimes going to the testChannelUpper and sometimes to the testChannelLower).
I basically end up with this across the executions :
(each time I am just hitting this endpoint http://localhost:9090/test?test=HellOoi)
Execution 1:
GenericMessage [payload=HELLOOI, headers={jobName=someActivity, history=someGateway,testGateway,testChannelUpper,testUpperFlow.channel#0, id=4aa7b075-23cc-6ab3-10a1-c7cb73bae49b, timestamp=1447686848477}]
Execution 2:
GenericMessage [payload=HELLOOI, headers={jobName=someActivity, history=someGateway,testGateway,testChannelUpper,testUpperFlow.channel#0, id=a18dcd01-da18-b00d-30c0-e1a03ce19104, timestamp=1447686853549}]
Execution 3:
GenericMessage [payload=hellooi, headers={jobName=someActivity, history=someGateway,testGateway,testChannelUpper,testLowerFlow.channel#0, id=5f0abcb9-378e-7a3c-9c93-a04ff6352927, timestamp=1447686857545}]
I believe that what I'm attempting here is also shown in the channelFlow example of the DSL wiki :
https://github.com/spring-projects/spring-integration-java-dsl/wiki/Spring-Integration-Java-DSL-Reference
Sooo the specs on what I'm using are :
Spring Boot v1.2.2.RELEASE
Spring v4.1.5.RELEASE
spring-integration-java-dsl 1.0.2.RELEASE
JDK 1.8.0_40-b25
So... has anyone else seen this kind of behavior? Am I just abusing the channel implementation? Any other ideas? Thanks in advance!
As Gary pointed out the best way to do this is to have a pub-sub and order the consumers on this :
#Bean(name = TEST_CHANNEL)
public MessageChannel testGatewayChannel() {
return MessageChannels.publishSubscribe(TEST_CHANNEL).get();
}
#Bean
public IntegrationFlow testUpperFlow() {
return IntegrationFlows
.from(TEST_CHANNEL)
.<String, String>transform(String::toUpperCase, e -> e.order(1))
.handle(System.out::println)
.get();
}
#Bean
public IntegrationFlow testLowerFlow() {
return IntegrationFlows
.from(TEST_CHANNEL)
.<String, String>transform(String::toLowerCase, e -> e.order(2))
.handle(System.out::println)
.get();
}
What is the purpose of this...
#Bean
public IntegrationFlow testFlow() {
return IntegrationFlows
.from(TEST_CHANNEL).fixedSubscriberChannel()
.channel(TEST_UPPER_CHANNEL)
.channel(TEST_LOWER_CHANNEL)
.get();
}
?
All that does is bridge the three channels together.
In fact, you end up with 2 consumers on TEST_UPPER_CHANNEL- the bridge in this flow and the transformer in your other flow.
By default, dispatching in direct channels uses round robin distribution. So the first message will go to the bridge, the next to the transformer, etc, etc.
I have a method which needs to execute multiple tasks async'ly.. I've managed to achieve that using the following IntegrationFlow:
#Bean
public IntegrationFlow startJobTask() {
return IntegrationFlows.from("TaskRoutingChannel")
.handle("jobService", "executeTasks")
.routeToRecipients(r -> r
.recipient("testTaskChannel")
.recipient("test2TaskChannel"))
.get();
}
#Bean ExecutorChannel testTaskChannel(){
return new ExecutorChannel(this.getAsyncExecutor());
}
#Bean ExecutorChannel test2TaskChannel(){
return new ExecutorChannel(this.getAsyncExecutor());
}
#Bean
public Executor getAsyncExecutor() {
SimpleAsyncTaskExecutor executor = new SimpleAsyncTaskExecutor();
return executor;
}
#Bean
public IntegrationFlow testTaskFlow() {
return IntegrationFlows.from("testTaskChannel")
.handle("testTaskService", "executeAsync")
.get();
}
#Bean
public IntegrationFlow test2TaskFlow() {
return IntegrationFlows.from("test2TaskChannel")
.handle("test2TaskService", "executeAsync")
.get();
}
The above flow is basically as follows:
TaskRoutingChannel calls a serviceActivator method executeTasks
executeTasks returns something like Message<List<myTask>>
this Message is routed to channels testTaskChannel and test2TaskChannel
which calls their own async serviceActivator methods.
Now, the issue is that I don't want to hardcode the recipient channels.
I could avoid hardcoding with normal router by setting the destination channel as a header. However, recepientListRouters don't seem to have the capability to get recipient channel name using expressions.
Is there any way to set the recipient channels dynamically?
Sorry for delay first of all.
Actually the regular .route() can do that for you, because it has exactly has this option:
protected abstract Collection<MessageChannel> determineTargetChannels(Message<?> message);
So, if your header extraction returns the list of channels, the HeaderValueRouter will be able to the message to all of them.
What is the best way to do validation in Spring Integration.
For example if we have an inbound gateway, when a message is received we want to validate it. If it's not valid -> return the validation errors to the gateway, else -> proceed with the normal flow of the application(transform, handle ...).
I tried a filter:
#Bean
public IntegrationFlow flow() {
return IntegrationFlows.from(requestChannel())
.transform(new MapToObjectTransformer(Campaign.class))
.filter(Campaign.class,
c -> c.getId() > 10 ? true : false, //if id > 10 then it's valid
e -> e.discardChannel(validationError()))
.handle(new MyHandler())
.get();
}
#Bean
public IntegrationFlow validationErrorFlow() {
return IntegrationFlows.from(validationError())
.handle(new ValidationHandler())//construct a message with the validation errors
.get();
}
It works, but that way if I use a spring validator then i have to call it twice, in the filter and in the ValidationHandler (can be a transformer) to get the errors.
Any better way?
.handle(new ValidationHandler())
You don't really need to create a new handler for each error.
In your filter, if the validation fails, throw MyValidationException(errors).
In the error flow on the gateway's error channel, the ErrorMessage has a payload that is a MessagingException with the MyValidatationException as its cause, and the failedMessage.
Something like...
.handle(validationErrorHandler())
...
#Bean
public MessageHandler validationErrorHandler() {
return new AbstractReplyProducingMessageHandler() {
public Object handleRequestMessage(Message<?> error) {
MyValidationException myEx = (MyValidationException)
((ErrorMessage) error).getPayload.getCause();
Errors errors = myEx.getErrors();
...
}
}
}
Or you can use a POJO messageHandler
public Object handle(MessagingException e) {
...
}