I am trying to configure a TCP socket that receives data in the format name,value in distinct messages. Those messages arrive on average every second, sometimes faster or sometimes slower.
I was able to set up a working configuration but I am lacking a basic understanding of what actually is happening in Spring Integration.
My configuration file looks like this:
#Configuration
#EnableIntegration
public class TCPSocketServerConfig
{
#Bean
public IntegrationFlow server(
final CSVProcessingService csvProcessingService,
#Value("${tcp.socket.server.port}") final int port
)
{
return IntegrationFlows.from(
Tcp.inboundAdapter(
Tcp.nioServer(port)
.deserializer(serializer())
.leaveOpen(true)
)
.autoStartup(true)
.outputChannel(queueChannel())
).transform(new ObjectToStringTransformer())
.handle(csvProcessingService)
.get();
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller()
{
return Pollers.fixedDelay(50, TimeUnit.MILLISECONDS).get();
}
#Bean
public MessageChannel queueChannel()
{
return MessageChannels.queue("queue", 50).get();
}
#Bean
public ByteArrayLfSerializer serializer()
{
final ByteArrayLfSerializer serializer = new ByteArrayLfSerializer();
serializer.setMaxMessageSize(10240);
return serializer;
}
}
And the CSVProcessingService looks like this (abbreviated):
#Slf4j
#Service
public class CSVProcessingService
{
#ServiceActivator
public void process(final String message)
{
log.debug("DATA RECEIVED: \n" + message);
final CsvMapper csvMapper = new CsvMapper();
final CsvSchema csvSchema = csvMapper.schemaFor(CSVParameter.class);
if (StringUtils.contains(message, StringUtils.LF))
{
processMultiLineInput(message, csvMapper, csvSchema);
}
else
{
processSingleLineInput(message, csvMapper, csvSchema);
}
}
}
My goals for this configuration are the following:
receive messages on the configured port
withstand a higher load without losing messages
deserialize the messages
put them into the queue channel
(ideally also log errors)
the queue channel is polled every 50 ms and the message from the queue channel passed to the ObjectToStringTransformer
after the transformer the converted message is passed to the CSVProcessingService for further processing
Did I achieve all those goals correctly or did I make a mistake because I misunderstood Spring Integration? Would it be possible to combine the Poller and the #ServiceActivator somehow?
Futhermore, I have a problem visualizing how my configured IntegrationFlow actually "flows", maybe somebody can help me to better understand this.
EDIT:
I reworked my configuration after Artems comment. It now look like this:
#Configuration
#EnableIntegration
public class TCPSocketServerConfig
{
#Value("${tcp.socket.server.port}") int port;
#Bean
public IntegrationFlow server(
final CSVProcessingService csvProcessingService
)
{
return IntegrationFlows.from(
Tcp.inboundAdapter(
tcpNioServer()
)
.autoStartup(true)
.errorChannel(errorChannel())
)
.transform(new ObjectToStringTransformer())
.handle(csvProcessingService)
.get();
}
#Bean
public AbstractServerConnectionFactory tcpNioServer()
{
return Tcp.nioServer(port)
.deserializer(serializer())
.leaveOpen(true)
.taskExecutor(
new ThreadPoolExecutor(0, 20,
30L, TimeUnit.SECONDS,
new SynchronousQueue<>(),
new DefaultThreadFactory("TCP-POOL"))
).get();
}
#Bean
public MessageChannel errorChannel()
{
return MessageChannels.direct("errors").get();
}
#Bean
public IntegrationFlow errorHandling()
{
return IntegrationFlows.from(errorChannel()).log(LoggingHandler.Level.DEBUG).get();
}
#Bean
public ByteArrayLfSerializer serializer()
{
final ByteArrayLfSerializer serializer = new ByteArrayLfSerializer();
serializer.setMaxMessageSize(10240);
return serializer;
}
}
I also removed the #ServiceActivator annotation form the CSVProcessingService#process method.
Not sure what confuses you, but your configuration and logic looks good.
You may miss the fact that you don't need a QueueChannel in between, since an AbstractConnectionFactory.processNioSelections() is already multi-threaded and it schedules a task to read a message from the socket. So, you only need is to configure an appropriate Executor for Tcp.nioServer(). Although it is an Executors.newCachedThreadPool() by default anyway.
On the other hand with in-memory QueueChannel you indeed may lose messages because they are already read from the network.
When you do Java DSL, you should consider to use poller() option on the endpoint. The #Poller will work on the #ServiceActivator if you have inputChannel attribute over there, but using the same in the handle() will override that inputChannel, so your #Poller won't be applied. Don't confuse yourself with mixing Java DSL and annotation configuration!
Everything else is good in your configuration.
Related
In my use case I need to do request-reply call to a remote system via managed queues. Using Spring Boot and IBM's MQ starter I have the problem that the application wants to create dynamic/temporary reply queues instead of using the already existing managed queue.
Configuration is set up here
#EnableJms
#Configuration
public class QueueConfiguration {
#Bean
public MQQueueConnectionFactory connectionFactory() throws JMSException {
MQQueueConnectionFactory factory = new MQQueueConnectionFactory();
factory.setTransportType(CT_WMQ); // is 1
factory.setHostName(queueProperties.getHost());
factory.setPort(queueProperties.getPort());
factory.setChannel(queueProperties.getChannel()); // combo of ${queueManager}%${channel}
return factory;
}
#Bean
public JmsMessagingTemplate messagingTemplate(ConnectionFactory connectionFactory) {
JmsMessagingTemplate jmt = new JmsMessagingTemplate(connectionFactory);
jmt.setDefaultDestinationName(queueProperties.getQueueName());
return jmt;
}
#Bean
public Jaxb2Marshaller marshaller() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setPackagesToScan("com.foo.model");
return marshaller;
}
#Bean
public MessageConverter messageConverter(Jaxb2Marshaller marshaller) {
MarshallingMessageConverter converter = new MarshallingMessageConverter();
converter.setMarshaller(marshaller);
converter.setUnmarshaller(marshaller);
return converter;
}
}
Usage is pretty straight forward: Take the object convert and send it. Wait for response receive
and convert it.
#Component
public class ExampleSenderReceiver {
#Autowired
private JmsMessagingTemplate jmsMessagingTemplate;
#Override
#SneakyThrows
public ResponseExample sendAndReceive(RequestExample request, String correlationId) {
MessagePostProcessor mpp = message -> {
message = MessageBuilder.fromMessage(message)
.setHeader(JmsHeaders.CORRELATION_ID, correlationId)
// .setHeader(JmsHeaders.REPLY_TO, "DEV.QUEUE.3") this triggers queue creation
.build();
return message;
};
String destination = Objects.requireNonNull(jmsMessagingTemplate.getDefaultDestinationName());
return jmsMessagingTemplate.convertSendAndReceive(destination, request, ResponseExample.class, mpp);
}
I read already a lot of IBM documentation and think, I need to set the message type to "MQMT_REQUEST" but I do not find the right spot to do so.
Update
Added Spring Integration as Gary proposed and added a configuration for JmsOutboundGateway
#Bean
public MessageChannel requestChannel() {
return new DirectChannel();
}
#Bean
public QueueChannel responseChannel() {
return new QueueChannel();
}
#Bean
#ServiceActivator(inputChannel = "requestChannel" )
public JmsOutboundGateway jmsOutboundGateway( ConnectionFactory connectionFactory) {
JmsOutboundGateway gateway = new JmsOutboundGateway();
gateway.setConnectionFactory(connectionFactory);
gateway.setRequestDestinationName("REQUEST");
gateway.setReplyDestinationName("RESPONSE");
gateway.setReplyChannel(responseChannel());
gateway.setCorrelationKey("JMSCorrelationID*");
gateway.setIdleReplyContainerTimeout(2, TimeUnit.SECONDS);
return gateway;
}
And adapted my ExampleSenderReceiver class
#Autowired
#Qualifier("requestChannel")
private MessageChannel requestChannel;
#Autowired
#Qualifier("responseChannel")
private QueueChannel responseChannel;
#Override
#SneakyThrows
public ResponseExample sendAndReceive(RequestExample request, String correlationId) {
String xmlContent = "the marshalled request object";
Map<String, Object> header = new HashMap<>();
header.put(JmsHeaders.CORRELATION_ID, correlationId);
GenericMessage<String> message1 = new GenericMessage<>(xmlContent, header);
requestChannel.send(message1);
log.info("send done" );
Message<?> receive = responseChannel.receive(1500);
if(null != receive){
log.info("incoming: {}", receive.toString());
}
}
The important part is gateway.setCorrelationKey("JMSCorrelationID*");
Without that line the correlationId was not propagated correct.
Next step is re-adding MessageConverters and make it nice again.
Thank you.
The default JmsTemplate (used by the JmsMessagingTemplate) always uses a temporary reply queue. You can subclass it and override doSendAndReceive(Session session, Destination destination, MessageCreator messageCreator) to use your managed queue instead.
However, it will only work if you have one request outstanding at a time (e.g. all run on a single thread). You will also have to add code for discarding "late" arrivals by checking the correlation id.
You can use async sends instead and handle replies on a listener container and correlate the replies to the requests.
Consider using spring-integration-jms and its outbound gateway instead - it has much more flexibility in reply queue handling (and does all the correlation for you).
https://docs.spring.io/spring-integration/reference/html/jms.html#jms-outbound-gateway
You are missing the queue manager.
ibm:
mq:
queueManager: QM1
channel: chanel
connName: localhost(1414)
user: admin
password: admin
I want to write an application wherein I need a producer sending messages using spring boot rabbitmq and the receiver is written in python. The receiver part was easy-
receive.py
#!/usr/bin/env python
import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.exchange_declare(exchange='topic_logs',
exchange_type='topic')
result = channel.queue_declare(exclusive=True)
queue_name = result.method.queue
binding_keys = sys.argv[1:]
if not binding_keys:
sys.stderr.write("Usage: %s [binding_key]...\n" % sys.argv[0])
sys.exit(1)
for binding_key in binding_keys:
channel.queue_bind(exchange='topic_logs',
queue=queue_name,
routing_key=binding_key)
print(' [*] Waiting for logs. To exit press CTRL+C')
def callback(ch, method, properties, body):
print(" [x] %r:%r" % (method.routing_key, body))
channel.basic_consume(callback,
queue=queue_name,
no_ack=True)
channel.start_consuming()
How to write a spring boot rabbitmq sender code for this? What are the necessary things to be declared there? Please help.
Spring Boot is Java application, first of all.
So, you need to make yourself familiar with that language.
And the choice is correct: you really can send to the RabbitMQ from Java and receive from the queue in any other client.
Spring Boot provides for you RabbitTemplate bean. So, if the story is about to send you just need to inject such a bean and use its API to send:
#Autowired
RabbitTemplate rabbitTemplate;
...
this.rabbitTemplate.convertAndSend("topic_logs", binding_key, data);
See Reference Manual for more info.
With spring integration, first you need to create a configuration class:
#Configuration
public class RabbitConfig {
#Autowired
private ConnectionFactory connectionFactory;
#Bean
DirectExchange dropfileExchange() {
return new DirectExchange("exchange_name", true, false);
}
#Bean
public Queue dropfileQueue() {
return new Queue("queue_name", true);
}
#Bean
Binding dropfileExchangeBinding(DirectExchange dropfileExchange, Queue dropfileQueue) {
return BindingBuilder.bind(dropfileQueue).to(dropfileExchange).with("key_name");
}
#Bean
public RabbitTemplate dropfileExchangeTemplate() {
RabbitTemplate rt = new RabbitTemplate(connectionFactory);
rt.setExchange("exchange_name");
rt.setRoutingKey("key_name");
rt.setConnectionFactory(connectionFactory);
return rt;
}
#Bean
public RabbitMessagingTemplate rabbitMessagingTemplate() {
return new RabbitMessagingTemplate(dropfileExchangeTemplate());
}
}
Then create a gateway service:
#MessagingGateway
public interface DropfileMessageGateway {
#Gateway(requestChannel = "channel_name")
void generate(String payload);
}
And then specify the producer flow using Java DSL as follows:
#Bean
public IntegrationFlow toOutboundQueueFlow() {
return IntegrationFlows.from("channel_name")
.transform(Transformers.toJson())
.handle(Amqp.outboundAdapter(rabbitConfig.dropfileExchangeTemplate())).get();
}
That will read a message from the channel, transform it to JSON and then dispatch it using an outbound adapter to the rabbit exchange.
Note that the messages go to the channel via the messaging gateway, so the channel name in both should be the same.
Do not forget to include the appropriate dependencies.
I may just be missing something very simple here (or misusing something), but I was attempting to set up two direct channels such that one flow would pass some data to each sequentially. So using the Spring Integration JAVA DSL I had something like this (significantly simplified for this example):
public static final String TEST_CHANNEL = "testGateway";
public static final String TEST_UPPER_CHANNEL = "testChannelUpper";
public static final String TEST_LOWER_CHANNEL = "testChannelLower";
#Bean(name = TEST_CHANNEL)
public MessageChannel testGatewayChannel() {
return MessageChannels.direct(TEST_CHANNEL).get();
}
#Bean(name = TEST_UPPER_CHANNEL)
public MessageChannel testChannelUpperChannel() {
return MessageChannels.direct(TEST_UPPER_CHANNEL).get();
}
#Bean(name = TEST_LOWER_CHANNEL)
public MessageChannel testChannelLowerChannel() {
return MessageChannels.direct(TEST_LOWER_CHANNEL).get();
}
#Bean
public IntegrationFlow testFlow() {
return IntegrationFlows
.from(TEST_CHANNEL)
.channel(TEST_UPPER_CHANNEL)
.channel(TEST_LOWER_CHANNEL)
.get();
}
#Bean
public IntegrationFlow testUpperFlow() {
return IntegrationFlows
.from(TEST_UPPER_CHANNEL)
.<String, String>transform(String::toUpperCase)
.handle(System.out::println)
.get();
}
#Bean
public IntegrationFlow testLowerFlow() {
return IntegrationFlows
.from(TEST_LOWER_CHANNEL)
.<String, String>transform(String::toLowerCase)
.handle(System.out::println)
.get();
}
I'm using a REST endpoint to invoke the flow via a gateway, but when I do so it seems only one of the channels is invoked. The channel also seems to be random across invocations (sometimes going to the testChannelUpper and sometimes to the testChannelLower).
I basically end up with this across the executions :
(each time I am just hitting this endpoint http://localhost:9090/test?test=HellOoi)
Execution 1:
GenericMessage [payload=HELLOOI, headers={jobName=someActivity, history=someGateway,testGateway,testChannelUpper,testUpperFlow.channel#0, id=4aa7b075-23cc-6ab3-10a1-c7cb73bae49b, timestamp=1447686848477}]
Execution 2:
GenericMessage [payload=HELLOOI, headers={jobName=someActivity, history=someGateway,testGateway,testChannelUpper,testUpperFlow.channel#0, id=a18dcd01-da18-b00d-30c0-e1a03ce19104, timestamp=1447686853549}]
Execution 3:
GenericMessage [payload=hellooi, headers={jobName=someActivity, history=someGateway,testGateway,testChannelUpper,testLowerFlow.channel#0, id=5f0abcb9-378e-7a3c-9c93-a04ff6352927, timestamp=1447686857545}]
I believe that what I'm attempting here is also shown in the channelFlow example of the DSL wiki :
https://github.com/spring-projects/spring-integration-java-dsl/wiki/Spring-Integration-Java-DSL-Reference
Sooo the specs on what I'm using are :
Spring Boot v1.2.2.RELEASE
Spring v4.1.5.RELEASE
spring-integration-java-dsl 1.0.2.RELEASE
JDK 1.8.0_40-b25
So... has anyone else seen this kind of behavior? Am I just abusing the channel implementation? Any other ideas? Thanks in advance!
As Gary pointed out the best way to do this is to have a pub-sub and order the consumers on this :
#Bean(name = TEST_CHANNEL)
public MessageChannel testGatewayChannel() {
return MessageChannels.publishSubscribe(TEST_CHANNEL).get();
}
#Bean
public IntegrationFlow testUpperFlow() {
return IntegrationFlows
.from(TEST_CHANNEL)
.<String, String>transform(String::toUpperCase, e -> e.order(1))
.handle(System.out::println)
.get();
}
#Bean
public IntegrationFlow testLowerFlow() {
return IntegrationFlows
.from(TEST_CHANNEL)
.<String, String>transform(String::toLowerCase, e -> e.order(2))
.handle(System.out::println)
.get();
}
What is the purpose of this...
#Bean
public IntegrationFlow testFlow() {
return IntegrationFlows
.from(TEST_CHANNEL).fixedSubscriberChannel()
.channel(TEST_UPPER_CHANNEL)
.channel(TEST_LOWER_CHANNEL)
.get();
}
?
All that does is bridge the three channels together.
In fact, you end up with 2 consumers on TEST_UPPER_CHANNEL- the bridge in this flow and the transformer in your other flow.
By default, dispatching in direct channels uses round robin distribution. So the first message will go to the bridge, the next to the transformer, etc, etc.
I have two separate applications running either side of an ActiveMQ broker; application 1 sends synchronous requests to application 2 which returns the response back to application 1. At present the replies are via temporary queues and I am now trying to create a named reply destination to avoid the overhead of creating multiple temporary queues.
Application 1
#MessagingGateway
public interface OrderGateway {
#Gateway(requestChannel = "requestChannel", replyChannel = "responseChannel")
public OrderDto fetchOrder(OrderRequest orderRequest);
}
#Bean
public IntegrationFlow outgoingRequestFlow(ConnectionFactory connectionFactory) {
return IntegrationFlows.from("requestChannel")
.handle(Jms.outboundGateway(connectionFactory)
.requestDestination("request.queue")
.replyDestination("response.topic")
.correlationKey("JMSCorrelationID"))
.channel("responseChannel")
.get();
}
Application 2
#Bean
public IntegrationFlow incomingRequestFlow(ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Jms.inboundGateway(connectionFactory)
.destination("request.queue")
.correlationKey("JMSCorrelationID"))
.channel("requestChannel")
.handle("requestServiceActivator", "handleRequest")
.channel("responseChannel")
.get();
}
#Component
public class OrderServiceActivator {
#Autowired
OrderService orderService;
#ServiceActivator
public OrderDto fetchOrder(OrderRequest orderRequest) {
return orderService.getById(orderRequest.getId());
}
}
When I start both applications request.queue gets created and has one consumer (application 2). response.topic gets created but for some reason it has no consumers. Consequently when I send a request in to application 1 it reaches application 2, but after 5 seconds application 1 does not receive a reply and times out and the following errors are logged:
Application 2
org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'org.springframework.web.context.WebApplicationContext:/application-2.responseChannel'
Application 1
org.springframework.integration.MessageTimeoutException: failed to receive JMS response within timeout of: 5000ms
I presume I've made some simple configuration error, any help would be appreciated.
With your configuration, there is no long-lived consumer for the reply queue - a consumer is created for each request (with a message selector for the specific correlation id).
If you add .replyContainer() there will be a permanent consumer.
However, it should make no difference functionally.
I just ran tests similar to yours with and without replyContainer() and it all worked fine for me...
#Bean
public IntegrationFlow jmsOutboundGatewayFlow() {
return f -> f.handleWithAdapter(a ->
a.jmsGateway(this.jmsConnectionFactory)
// .replyContainer()
.replyDestination("pipereplies")
.correlationKey("JmsCorrelationID")
.requestDestination("jmsPipelineTest"));
}
I suggest you turn on debug logging to see if that sheds some light.
#Bean
public IntegrationFlow jmsInboundGatewayFlow() {
return IntegrationFlows.from((MessagingGateways g) ->
g.jms(this.jmsConnectionFactory)
.correlationKey("JmsCorrelationID")
.destination("jmsPipelineTest"))
.<String, String>transform(String::toUpperCase)
.get();
}
I'm failing to find how the #ServiceActivator is getting wired...
Usually it's something like:
#ServiceActivator(inputChannel = "requestChannel", outputChannel = "responseChannel")
public .....
Perhaps that is what you are missing.
I have a method which needs to execute multiple tasks async'ly.. I've managed to achieve that using the following IntegrationFlow:
#Bean
public IntegrationFlow startJobTask() {
return IntegrationFlows.from("TaskRoutingChannel")
.handle("jobService", "executeTasks")
.routeToRecipients(r -> r
.recipient("testTaskChannel")
.recipient("test2TaskChannel"))
.get();
}
#Bean ExecutorChannel testTaskChannel(){
return new ExecutorChannel(this.getAsyncExecutor());
}
#Bean ExecutorChannel test2TaskChannel(){
return new ExecutorChannel(this.getAsyncExecutor());
}
#Bean
public Executor getAsyncExecutor() {
SimpleAsyncTaskExecutor executor = new SimpleAsyncTaskExecutor();
return executor;
}
#Bean
public IntegrationFlow testTaskFlow() {
return IntegrationFlows.from("testTaskChannel")
.handle("testTaskService", "executeAsync")
.get();
}
#Bean
public IntegrationFlow test2TaskFlow() {
return IntegrationFlows.from("test2TaskChannel")
.handle("test2TaskService", "executeAsync")
.get();
}
The above flow is basically as follows:
TaskRoutingChannel calls a serviceActivator method executeTasks
executeTasks returns something like Message<List<myTask>>
this Message is routed to channels testTaskChannel and test2TaskChannel
which calls their own async serviceActivator methods.
Now, the issue is that I don't want to hardcode the recipient channels.
I could avoid hardcoding with normal router by setting the destination channel as a header. However, recepientListRouters don't seem to have the capability to get recipient channel name using expressions.
Is there any way to set the recipient channels dynamically?
Sorry for delay first of all.
Actually the regular .route() can do that for you, because it has exactly has this option:
protected abstract Collection<MessageChannel> determineTargetChannels(Message<?> message);
So, if your header extraction returns the list of channels, the HeaderValueRouter will be able to the message to all of them.