I'm working on an integration with a REST service, the idea is that it's polled by an outbound gateway marketingCategoryOutboundGateway implemented by HttpRequestExecutingMessageHandler. The gateway makes a request to the REST service and pushes its response to the marketingCategory channel. The gateway itself is triggered by a message created by marketingCategoryPollerMessageSource using the makeTriggeringMessage factory method.
The problem is that the service returns paginated results. I something which would listen on the marketingCategory channel, apart from the service activator I already have, check if the response and push a new message with an incremented page number created by makeTriggeringMessage to the marketingCategoryPoller channel, so that the code would spin in a loop until it fetches all the pages from the REST service.
Does Spring Integration allow to make such filters which receive one message on the input channel, test it against a condition and push a new message to the output channel if the condition is true?
The code:
//Responses from the REST service go to this channel
#Bean("marketingCategory")
MessageChannel marketingCategory() { return new PublishSubscribeChannel();}
//This channel is used to trigger the outbound gateway which makes a request to the REST service
#Bean
MessageChannel marketingCategoryPoller() {return new DirectChannel();}
//An adapter creating triggering messages for the gateway
#Bean
#InboundChannelAdapter(channel = "marketingCategoryPoller", poller = #Poller(fixedDelay = "15000"))
public MessageSource<String> marketingCategoryPollerMessageSource() { return () -> makeTriggeringMessage(1);}
//A factory for producing messages which trigger the gateway
private Message<String> makeTriggeringMessage(int page) {
//make a message for triggering marketingCategoryOutboundGateway
return MessageBuilder.withPayload("")
.setHeader("Host", "eclinic")
.setHeader("page", page)
.build();
}
//An outbound gateway, makes a request to the REST service and returns the response to marketingCategory channel
#Bean
#ServiceActivator(inputChannel = "marketingCategoryPoller")
public MessageHandler marketingCategoryOutboundGateway(#Qualifier("marketingCategory") MessageChannel channel) {
//make a request to the REST service and push the response to the marketingCategory channel
}
//handler for REST service responses
#Bean
#ServiceActivator(inputChannel = "marketingCategory")
public MessageHandler marketingCategoryHandler() {
return (msg) -> {
//process the categories returned by marketingCategoryOutboundGateway
};
}
I've found a solution based on this posting Read and download from a paginated REST-Services with spring integration:
Trigger the outbound gateway which talks to the REST service and pushes the response to a channel using an inbound channel adapter with a poller.
The inbound channel adapter is a message source which originally generates a message with a header indicating the page number to fetch from the REST API.
The page message header is used by the outbound gateway to generate a url specifying the desired page
The channel to which the outbound gateway pushes REST service responses has 2 subscribers:
2.1. a service activator which does something with the fetched data
2.2. a filter, which checks if this is the last page and if not, it sends the message further to another channel used by a header enricher
Having received a message, the header enricher increments its page header and pushes the message further to the channel which triggers the outbound gateway,
the gateway read the incremented page header and fetches the next page from the REST service
The loop keeps spinning until the REST service returns the last page. The filter doesn't let this message to pass through to the header enricher breaking the loop.
Full code:
#Configuration
public class IntegrationConfiguration {
private final ApiGateConfig apiGateConfig;
IntegrationConfiguration(ApiGateConfig apiGateConfig) {
this.apiGateConfig = apiGateConfig;
}
#Bean("marketingCategory")
MessageChannel marketingCategory() {
return new PublishSubscribeChannel();
}
#Bean
MessageChannel marketingCategoryPoller() {
return new DirectChannel();
}
#Bean
MessageChannel marketingCategoryPollerNextPage() {
return new DirectChannel();
}
#Bean
#InboundChannelAdapter(channel = "marketingCategoryPoller", poller = #Poller(fixedDelay = "15000"))
public MessageSource<RestPageImpl<MarketingCategory>> marketingCategoryPollerMessageSource() {
return () -> makeTriggeringMessage(0);
}
/**
* Build a gateway triggering message
*/
private Message<RestPageImpl<MarketingCategory>> makeTriggeringMessage(int page) {
return MessageBuilder.withPayload(new RestPageImpl<MarketingCategory>())
.setHeader("Host", "eclinic")
.setHeader("page", page)
.build();
}
#Bean
#ServiceActivator(inputChannel = "marketingCategoryPoller")
public MessageHandler marketingCategoryOutboundGateway(#Qualifier("marketingCategory") MessageChannel channel) {
String uri = apiGateConfig.getUri() + "/marketingCategories?page={page}";
//the type of the payload
ParameterizedTypeReference<RestPageImpl<MarketingCategory>> type = new ParameterizedTypeReference<>() {
};
//page number comes from the message
SpelExpressionParser expressionParser = new SpelExpressionParser();
var uriVariables = new HashMap<String, Expression>();
uriVariables.put("page", expressionParser.parseExpression("headers.page"));
HttpRequestExecutingMessageHandler handler = new HttpRequestExecutingMessageHandler(uri);
handler.setHttpMethod(HttpMethod.GET);
handler.setExpectedResponseTypeExpression(new ValueExpression<>(type));
handler.setOutputChannel(channel);
handler.setUriVariableExpressions(uriVariables);
return handler;
}
#Bean
#ServiceActivator(inputChannel = "marketingCategory")
public MessageHandler marketingCategoryHandler() {
return (msg) -> {
var page = (RestPageImpl<MarketingCategory>) msg.getPayload();
System.out.println("Page #" + page.getNumber());
page.getContent().forEach(c -> System.out.println(c.getMarketingCategory()));
};
}
#Filter(inputChannel = "marketingCategory", outputChannel = "marketingCategoryPollerNextPage")
public boolean marketingCategoryPaginationFilter(RestPageImpl<MarketingCategory> page) {
return !page.isLast();
}
#Bean
#Transformer(inputChannel = "marketingCategoryPollerNextPage", outputChannel = "marketingCategoryPoller")
HeaderEnricher incrementPage() {
Map<String, HeaderValueMessageProcessor<?>> headersToAdd = new HashMap<>();
Expression expression = new SpelExpressionParser().parseExpression("headers.page+1");
var valueProcessor = new ExpressionEvaluatingHeaderValueMessageProcessor<>(expression, Integer.class);
valueProcessor.setOverwrite(true);
headersToAdd.put("page", valueProcessor);
return new HeaderEnricher(headersToAdd);
}
}
Related
In my use case I need to do request-reply call to a remote system via managed queues. Using Spring Boot and IBM's MQ starter I have the problem that the application wants to create dynamic/temporary reply queues instead of using the already existing managed queue.
Configuration is set up here
#EnableJms
#Configuration
public class QueueConfiguration {
#Bean
public MQQueueConnectionFactory connectionFactory() throws JMSException {
MQQueueConnectionFactory factory = new MQQueueConnectionFactory();
factory.setTransportType(CT_WMQ); // is 1
factory.setHostName(queueProperties.getHost());
factory.setPort(queueProperties.getPort());
factory.setChannel(queueProperties.getChannel()); // combo of ${queueManager}%${channel}
return factory;
}
#Bean
public JmsMessagingTemplate messagingTemplate(ConnectionFactory connectionFactory) {
JmsMessagingTemplate jmt = new JmsMessagingTemplate(connectionFactory);
jmt.setDefaultDestinationName(queueProperties.getQueueName());
return jmt;
}
#Bean
public Jaxb2Marshaller marshaller() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setPackagesToScan("com.foo.model");
return marshaller;
}
#Bean
public MessageConverter messageConverter(Jaxb2Marshaller marshaller) {
MarshallingMessageConverter converter = new MarshallingMessageConverter();
converter.setMarshaller(marshaller);
converter.setUnmarshaller(marshaller);
return converter;
}
}
Usage is pretty straight forward: Take the object convert and send it. Wait for response receive
and convert it.
#Component
public class ExampleSenderReceiver {
#Autowired
private JmsMessagingTemplate jmsMessagingTemplate;
#Override
#SneakyThrows
public ResponseExample sendAndReceive(RequestExample request, String correlationId) {
MessagePostProcessor mpp = message -> {
message = MessageBuilder.fromMessage(message)
.setHeader(JmsHeaders.CORRELATION_ID, correlationId)
// .setHeader(JmsHeaders.REPLY_TO, "DEV.QUEUE.3") this triggers queue creation
.build();
return message;
};
String destination = Objects.requireNonNull(jmsMessagingTemplate.getDefaultDestinationName());
return jmsMessagingTemplate.convertSendAndReceive(destination, request, ResponseExample.class, mpp);
}
I read already a lot of IBM documentation and think, I need to set the message type to "MQMT_REQUEST" but I do not find the right spot to do so.
Update
Added Spring Integration as Gary proposed and added a configuration for JmsOutboundGateway
#Bean
public MessageChannel requestChannel() {
return new DirectChannel();
}
#Bean
public QueueChannel responseChannel() {
return new QueueChannel();
}
#Bean
#ServiceActivator(inputChannel = "requestChannel" )
public JmsOutboundGateway jmsOutboundGateway( ConnectionFactory connectionFactory) {
JmsOutboundGateway gateway = new JmsOutboundGateway();
gateway.setConnectionFactory(connectionFactory);
gateway.setRequestDestinationName("REQUEST");
gateway.setReplyDestinationName("RESPONSE");
gateway.setReplyChannel(responseChannel());
gateway.setCorrelationKey("JMSCorrelationID*");
gateway.setIdleReplyContainerTimeout(2, TimeUnit.SECONDS);
return gateway;
}
And adapted my ExampleSenderReceiver class
#Autowired
#Qualifier("requestChannel")
private MessageChannel requestChannel;
#Autowired
#Qualifier("responseChannel")
private QueueChannel responseChannel;
#Override
#SneakyThrows
public ResponseExample sendAndReceive(RequestExample request, String correlationId) {
String xmlContent = "the marshalled request object";
Map<String, Object> header = new HashMap<>();
header.put(JmsHeaders.CORRELATION_ID, correlationId);
GenericMessage<String> message1 = new GenericMessage<>(xmlContent, header);
requestChannel.send(message1);
log.info("send done" );
Message<?> receive = responseChannel.receive(1500);
if(null != receive){
log.info("incoming: {}", receive.toString());
}
}
The important part is gateway.setCorrelationKey("JMSCorrelationID*");
Without that line the correlationId was not propagated correct.
Next step is re-adding MessageConverters and make it nice again.
Thank you.
The default JmsTemplate (used by the JmsMessagingTemplate) always uses a temporary reply queue. You can subclass it and override doSendAndReceive(Session session, Destination destination, MessageCreator messageCreator) to use your managed queue instead.
However, it will only work if you have one request outstanding at a time (e.g. all run on a single thread). You will also have to add code for discarding "late" arrivals by checking the correlation id.
You can use async sends instead and handle replies on a listener container and correlate the replies to the requests.
Consider using spring-integration-jms and its outbound gateway instead - it has much more flexibility in reply queue handling (and does all the correlation for you).
https://docs.spring.io/spring-integration/reference/html/jms.html#jms-outbound-gateway
You are missing the queue manager.
ibm:
mq:
queueManager: QM1
channel: chanel
connName: localhost(1414)
user: admin
password: admin
I'm trying to send the UDP request and receive the response. Spring Integration has the appropriate instruments for such kind of task: UnicastSendingMessageHandler and UnicastReceivingChannelAdapter. I configured it in the following way
#Bean
public MessageChannel requestChannel() {
return new DirectChannel();
}
#Bean
#ServiceActivator(inputChannel = "requestChannel")
public UnicastSendingMessageHandler unicastSendingMessageHandler() {
UnicastSendingMessageHandler unicastSendingMessageHandler = new UnicastSendingMessageHandler("239.255.255.250", 1982);
return unicastSendingMessageHandler;
}
#Bean
public UnicastReceivingChannelAdapter unicastReceivingChannelAdapter() {
UnicastReceivingChannelAdapter unicastReceivingChannelAdapter = new UnicastReceivingChannelAdapter(8080);
unicastReceivingChannelAdapter.setOutputChannelName("nullChannel");
return unicastReceivingChannelAdapter;
}
How I send a message (I'm using sendDiscoveryMessage() wherever I want):
#Service
public class DiscoveryService {
private static final String DISCOVERY_MESSAGE = "M-SEARCH * HTTP/1.1\r\n"
+ "HOST: 239.255.255.250:1982\r\n"
+ "MAN: \"ssdp:discover\"\r\n"
+ "ST: wifi_bulb";
private final MessageChannel requestChannel;
public DiscoveryService(final MessageChannel requestChannel) {
this.requestChannel = requestChannel;
}
public void sendDiscoveryMessage() {
requestChannel.send(new GenericMessage<>(DISCOVERY_MESSAGE));
}
}
At this point, I can check the packets via WireShark and ensure that Datagram was sent and the appropriate response was sent too.
The only question is how to receive this response. As far as I understand reading the documentation, I need the method annotated with #ServiceActivator. But I don't understand where (which channel) I should receive the response (in order to correctly specify #ServiceActivator(inputChannel="")). Also, I'm not sure about #ServiceActivator(inputChannel = "requestChannel") I put for UnicastSendingMessageHandler bean.
I tried to create the following method(assuming that the response will come to the same channel):
#ServiceActivator(inputChannel = "requestChannel")
public void receiveResponse(Message<String> response) {
System.out.println(response);
}
but it actually intercepts my own request message (seems logical to me, because I send the request to requestChannel).
So I don't understand how many channels I need (maybe I need 1 for request and 1 for response) and how to create #ServiceActivator to catch the response.
unicastReceivingChannelAdapter.setOutputChannelName("nullChannel");
You are sending the result to nullChannel which is like /dev/null on Unix; you are discarding it.
Use #ServiceActivator(inputChannel = "replyChannel") and
unicastReceivingChannelAdapter.setOutputChannelName("replyChannel");
I currently have a TcpInboundGateway that takes in messages, does some processing on the message and then returns the appropriate response, all as a TcpInboundGateway should.
However, I am curious if this TcpInboundGateway can be configured in such a way that it will send an immediate response to the originating request but continue to process the request and send the post-processing response as well?
Think of this immediate response as an acknowledgement to the sender that the message was received.
Possible Solution:
After reviewing this post, I came up with what I believe to be a viable solution to this problem.
#Configuration
#EnableIntegration
public class Configuration {
#Bean
public AbstractServerConnectionFactory serverConnectionFactory() {
return new TcpNetServerConnectionFactory(2002);
}
#Bean
public TcpReceivingChannelAdapter inboundAdapter(AbstractServerConnectionFactory serverConnectionFactory) {
TcpReceivingChannelAdapter inboundAdapter = new TcpReceivingChannelAdapter();
inboundAdapter.setConnectionFactory(serverConnectionFactory);
inboundAdapter.setOutputChannelName("sendAcknowledgement");
return inboundAdapter;
}
#MessageEndpoint
public class InboundMessageHandler {
#Autowired
private OutboundMessageGateway gateway;
#ServiceActivator(inputChannel="sendAcknowledgement", outputChannel="doProcessing")
public Message<String> initialAck(Message<String> message) {
gateway.send("ACK", message.getHeaders().get(IpHeaders.CONNECTION_ID).toString());
return message;
}
#ServiceActivator(inputChannel="doProcessing", outputChannel="sendResponse")
public Message<String> mockDelay(Message<String> message) throws InterruptedException {
return message;
}
}
#MessagingGateway(defaultRequestChannel="sendResponse")
public interface OutboundMessageGateway {
void send(#Payload String message, #Header(IpHeaders.CONNECTION_ID) String connectionId);
}
#Bean
#ServiceActivator(inputChannel="sendResponse")
public TcpSendingMessageHandler outboundAdapter(AbstractServerConnectionFactory serverConnectionFactory) {
TcpSendingMessageHandler outboundAdapter = new TcpSendingMessageHandler();
outboundAdapter.setConnectionFactory(serverConnectionFactory);
return outboundAdapter;
}
}
For the use-case with the TcpInboundGateway and acking with later reply you need to use a PublishSubscribeChannel with an Executor injected to make a processing async.
The first subscriber should return some ack into the replyChannel header. This way your TcpInboundGateway will perform request-reply and return that ack into the socket connected.
At the same time as you want, the second subscriber can perform desired logic and build the real reply later. Only the point that we need to use the mention in the docs Collaborating Outbound and Inbound Channel Adapters (as you noticed already). So, since TcpInboundGateway populates an IpHeaders.CONNECTION_ID header into a request message, it is going to be available in your async process and subsequent TcpSendingMessageHandler will know where to send your processed reply:
private void handleMessageAsServer(Message<?> message) {
// We don't own the connection, we are asynchronously replying
String connectionId = message.getHeaders().get(IpHeaders.CONNECTION_ID, String.class);
TcpConnection connection = null;
if (connectionId != null) {
connection = this.connections.get(connectionId);
}
if (connection != null) {
try {
connection.send(message);
}
So, what you need is like this:
a PublishSubscribeChannel with an executor for your TcpInboundGateway
A simple handler to reply with an ack as a first subscriber
A sub-flow for processing a request
A TcpSendingMessageHandler to send a process response into the same TCP connection.
Using Spring-Integration-Kafka can we still use #MessagingGateway and #Gateway.
My current code looks like this:
#MessagingGateway
public interface OrderGateway {
#Gateway(requestChannel = "requestChannel", replyChannel = "replyChannel",headers = {#GatewayHeader(name = "kafka_topic", value ="requestTopic"))
Order order(Item item)
}
on my Spring Spring configuration:
#Bean
#ServiceActivator(inputChannel = "requestChannel")
public MessageHandler kafkaMessageHandler(KafkaTemplate kafkaTemplate) {
KafkaProducerMessageHandler<String, String> messageHandler = new KafkaProducerMessageHandler<>(kafkaTemplate);
messageHandler.setMessageKeyExpression(new LiteralExpression("spring-integration-kafka"));
messageHandler.setTopicExpression(new SpelExpressionParser().parseExpression("headers.kafka_topic"));
return messageHandler;
}
with this setup I get and error saying:
by: org.springframework.messaging.core.DestinationResolutionException: no output-channel or replyChannel header available
You don't appear to have shown the complete configuration.
The gateway is expecting a reply but the kafkaMessageHandler produces no reply (unless the template is a ReplyingKafkaTemplate) and will lose the replyChannel header.
So, presumably, you are trying to send a reply from someplace else.
If you are expecting request/reply semantics; use the new outbound gateway.
Where the template has to be a ReplyingKafkaTemplate.
I wrote a simple message flow with request and reply. I have to use two independent queues so i declare AmqpOutboundAdapter to send a message and AmqpInboundAdapter to receive a reply.
#Bean
#FindADUsers
public AmqpOutboundEndpoint newFindADUsersOutboundAdapter() {
return Amqp.outboundAdapter(amqpTemplate())
.routingKeyExpression("headers[" + ADUsersFindConfig.ROUTING_KEY_HEADER + "]")
.exchangeName(getExchange())
.headerMapper(amqpHeaderMapper())
.get();
}
#Bean
public AmqpInboundChannelAdapter newFindADUsersResponseInboundChannelAdapter(
ADUsersFindResponseConfig config) {
return Amqp.inboundAdapter(rabbitConnectionFactory(), findADUsersResponseQueue)
.headerMapper(amqpHeaderMapper())
.outputChannel(config.newADUsersFindResponseOutputChannel())
.get();
}
It should work with #MessagingGateway:
#MessagingGateway
public interface ADUsersFindService {
String FIND_AD_USERS_CHANNEL = "adUsersFindChannel";
String FIND_AD_USERS_REPLY_OUTPUT_CHANNEL = "adUsersFindReplyOutputChannel";
String FIND_AD_USERS_REPLY_CHANNEL = "adUsersFindReplyChannel";
String CORRELATION_ID_REQUEST_HEADER = "correlation_id";
String ROUTING_KEY_HEADER = "replyRoutingKey";
String OBJECT_TYPE_HEADER = "object.type";
#Gateway(requestChannel = FIND_AD_USERS_CHANNEL, replyChannel = FIND_AD_USERS_REPLY_CHANNEL)
ADResponse find(ADRequest adRequest, #Header(ROUTING_KEY_HEADER) String routingKey, #Header(OBJECT_TYPE_HEADER) String objectType);
}
And the ADUsersFindResponseConfig class looks like:
#Configuration
#Import(JsonConfig.class)
public class ADUsersFindResponseConfig {
#Autowired
public NullChannel nullChannel;
#Autowired
private JsonObjectMapper<?, ?> mapper;
/**
* #return The output channel for the flow
*/
#Bean(name = ADUsersFindService.FIND_AD_USERS_REPLY_OUTPUT_CHANNEL)
public MessageChannel newADUsersFindResponseOutputChannel() {
return MessageChannels.direct().get();
}
/**
* #return The output channel for gateway
*/
#Bean(name = ADUsersFindService.FIND_AD_USERS_REPLY_CHANNEL)
public MessageChannel newADUsersFindResponseChannel() {
return MessageChannels.direct().get();
}
#Bean
public IntegrationFlow findADUsersResponseFlow() {
return IntegrationFlows
.from(newADUsersFindResponseOutputChannel())
.transform(new JsonToObjectTransformer(ADResponse.class, mapper))
.channel(newADUsersFindResponseChannel())
.get();
}
}
Sending message works properly, but i have a problem with receiving message. I am expecting that received message will be passed to channel called FIND_AD_USERS_REPLY_OUTPUT_CHANNEL, then the message will be deserialized to ADResponse object using findADUsersResponseFlow , and next ADResponse object will be passed to gateway replyChannel - FIND_AD_USERS_REPLY_CHANNEL. Finally, 'find' method return this object. Unfortunately when org.springframework.integration.handler.BridgeHandler receive a message, i got exception:
org.springframework.messaging.MessagingException: ; nested exception is org.springframework.messaging.core.DestinationResolutionException: no output-channel or replyChannel header available
Message log looks like:
11:51:35.697 [SimpleAsyncTaskExecutor-1] INFO New message - GenericMessage [payload={...somepayload...}, headers={correlation_id=7cbd958e-4b09-4e4c-ba8e-5ba574f3309a, replyRoutingKey=findADUsersResponse.ad, amqp_consumerQueue=findADUsersResponseQueue, history=newFindADUsersResponseInboundChannelAdapter,adUsersFindReplyOutputChannel,adUsersFindReplyChannel,infoLog,infoLoggerChain.channel#0,infoLoggerChain.channel#1, id=37a4735d-6983-d1ad-e0a1-b37dc17e48ef, amqp_consumerTag=amq.ctag-8Qs5YEun1jXYRf85Hu1URA, object.type=USER, timestamp=1469094695697}]
So i'm pretty sure that message was passed to adUsersFindReplyChannel. Also (if it's important) both request message and reply message have 'replyTo' header set to null. What am I doing wrong?
The replyChannel header is a live object and can't be serialized over AMQP.
You can use an outbound gateway instead of the pair of adapters and the framework will take care of the headers.
If you must use adapters for some reason, you need to do 2 things:
Use the header channel registry to convert the channel object to a String which is registered with the registry.
Make sure that the header mapper is configured to send/receive the replyChannel header and that your receiving system returns the header in the reply.