Develop Reference

Spring Boot App not consuming queue messages

I have the Rabbit MQ broker for communicating asynchronously between services. Service A is sending messages to the queue. I checked the queue and the messages from Service A have arrived:
I am trying to create a listener in the Service B in order to consume the messages produced by Service A. I verified like below to check if Service B is connected with RabbitMQ and it seems to be connected successfully.
The problem is that Service B started successfully but it is receiving messages from Rabbit MQ.
Below is the implementation of the listener:
#Slf4j
#Component
public class EventListener {
public static final String QUEUE_NAME = "events";
#RabbitListener(
bindings = {
#QueueBinding(
value = #Queue(QUEUE_NAME),
exchange = #Exchange("exchange")
)
}
)
public void handleTaskPayload(#Payload String payload) {
System.out.println(payload);
}
}
I verified the queue and exchange information in the Rabbit MQ and they are correct.
Everything is working correctly and there is no error thrown in service A or service B which makes this problem much harder to debug.
I tried to retrieve the message from the queue getMessage of RabbitMQ the message is like the below:
{"id":"1",:"name:"Test","created":null}
I will appreciate any help or guidance towards the solution of this problem.
Best Regards,
Rando.
P.S
I created a new test queue like the below and published some messages:
Modified the listener code like below and still wasn't able to trigger listener to listen to the queue events:
#Slf4j
#Component
public class RobotRunEventListener {
public static final String QUEUE_NAME = "test";
#RabbitListener(
bindings = {
#QueueBinding(
value = #Queue(QUEUE_NAME),
key = "test",
exchange = #Exchange("default")
)
}
)
public void handleTaskPayload(#Payload String payload) {
System.out.println(payload);
}

Try this approach:
#RabbitListener(queues = "test")
public void receive(String in, #Headers Map<String, Object> headers) throws IOException {
}

The problem was that the spring boot app that I was working on had a #Conditional(Config.class) that prevented the creation of the bean below:
#Slf4j
#Conditional(Config.class)
#EnableRabbit
public class InternalRabbitBootstrapConfiguration {
#Bean
public RabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setMaxConcurrentConsumers(5);
return factory;
}
...
which resulted in the spring boot app not listening to Rabbit MQ events. The Config.class required a specific profile in order to enable the app to listen to Rabbit MQ events.
public class DexiModeCondition implements Condition {
#Override
public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {
String[] activeProfiles = context.getEnvironment().getActiveProfiles();
return activeProfiles[0].equalsIgnoreCase(mode);
}
}

How to properly configure a TCP inboundAdapter with QueueChannel and ServiceActivator

I am trying to configure a TCP socket that receives data in the format name,value in distinct messages. Those messages arrive on average every second, sometimes faster or sometimes slower.
I was able to set up a working configuration but I am lacking a basic understanding of what actually is happening in Spring Integration.
My configuration file looks like this:
#Configuration
#EnableIntegration
public class TCPSocketServerConfig
{
#Bean
public IntegrationFlow server(
final CSVProcessingService csvProcessingService,
#Value("${tcp.socket.server.port}") final int port
)
{
return IntegrationFlows.from(
Tcp.inboundAdapter(
Tcp.nioServer(port)
.deserializer(serializer())
.leaveOpen(true)
)
.autoStartup(true)
.outputChannel(queueChannel())
).transform(new ObjectToStringTransformer())
.handle(csvProcessingService)
.get();
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller()
{
return Pollers.fixedDelay(50, TimeUnit.MILLISECONDS).get();
}
#Bean
public MessageChannel queueChannel()
{
return MessageChannels.queue("queue", 50).get();
}
#Bean
public ByteArrayLfSerializer serializer()
{
final ByteArrayLfSerializer serializer = new ByteArrayLfSerializer();
serializer.setMaxMessageSize(10240);
return serializer;
}
}
And the CSVProcessingService looks like this (abbreviated):
#Slf4j
#Service
public class CSVProcessingService
{
#ServiceActivator
public void process(final String message)
{
log.debug("DATA RECEIVED: \n" + message);
final CsvMapper csvMapper = new CsvMapper();
final CsvSchema csvSchema = csvMapper.schemaFor(CSVParameter.class);
if (StringUtils.contains(message, StringUtils.LF))
{
processMultiLineInput(message, csvMapper, csvSchema);
}
else
{
processSingleLineInput(message, csvMapper, csvSchema);
}
}
}
My goals for this configuration are the following:
receive messages on the configured port
withstand a higher load without losing messages
deserialize the messages
put them into the queue channel
(ideally also log errors)
the queue channel is polled every 50 ms and the message from the queue channel passed to the ObjectToStringTransformer
after the transformer the converted message is passed to the CSVProcessingService for further processing
Did I achieve all those goals correctly or did I make a mistake because I misunderstood Spring Integration? Would it be possible to combine the Poller and the #ServiceActivator somehow?
Futhermore, I have a problem visualizing how my configured IntegrationFlow actually "flows", maybe somebody can help me to better understand this.
EDIT:
I reworked my configuration after Artems comment. It now look like this:
#Configuration
#EnableIntegration
public class TCPSocketServerConfig
{
#Value("${tcp.socket.server.port}") int port;
#Bean
public IntegrationFlow server(
final CSVProcessingService csvProcessingService
)
{
return IntegrationFlows.from(
Tcp.inboundAdapter(
tcpNioServer()
)
.autoStartup(true)
.errorChannel(errorChannel())
)
.transform(new ObjectToStringTransformer())
.handle(csvProcessingService)
.get();
}
#Bean
public AbstractServerConnectionFactory tcpNioServer()
{
return Tcp.nioServer(port)
.deserializer(serializer())
.leaveOpen(true)
.taskExecutor(
new ThreadPoolExecutor(0, 20,
30L, TimeUnit.SECONDS,
new SynchronousQueue<>(),
new DefaultThreadFactory("TCP-POOL"))
).get();
}
#Bean
public MessageChannel errorChannel()
{
return MessageChannels.direct("errors").get();
}
#Bean
public IntegrationFlow errorHandling()
{
return IntegrationFlows.from(errorChannel()).log(LoggingHandler.Level.DEBUG).get();
}
#Bean
public ByteArrayLfSerializer serializer()
{
final ByteArrayLfSerializer serializer = new ByteArrayLfSerializer();
serializer.setMaxMessageSize(10240);
return serializer;
}
}
I also removed the #ServiceActivator annotation form the CSVProcessingService#process method.

Not sure what confuses you, but your configuration and logic looks good.
You may miss the fact that you don't need a QueueChannel in between, since an AbstractConnectionFactory.processNioSelections() is already multi-threaded and it schedules a task to read a message from the socket. So, you only need is to configure an appropriate Executor for Tcp.nioServer(). Although it is an Executors.newCachedThreadPool() by default anyway.
On the other hand with in-memory QueueChannel you indeed may lose messages because they are already read from the network.
When you do Java DSL, you should consider to use poller() option on the endpoint. The #Poller will work on the #ServiceActivator if you have inputChannel attribute over there, but using the same in the handle() will override that inputChannel, so your #Poller won't be applied. Don't confuse yourself with mixing Java DSL and annotation configuration!
Everything else is good in your configuration.

How do I write a spring boot rabbitmq sender for a python receiver?

I want to write an application wherein I need a producer sending messages using spring boot rabbitmq and the receiver is written in python. The receiver part was easy-
receive.py
#!/usr/bin/env python
import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.exchange_declare(exchange='topic_logs',
exchange_type='topic')
result = channel.queue_declare(exclusive=True)
queue_name = result.method.queue
binding_keys = sys.argv[1:]
if not binding_keys:
sys.stderr.write("Usage: %s [binding_key]...\n" % sys.argv[0])
sys.exit(1)
for binding_key in binding_keys:
channel.queue_bind(exchange='topic_logs',
queue=queue_name,
routing_key=binding_key)
print(' [*] Waiting for logs. To exit press CTRL+C')
def callback(ch, method, properties, body):
print(" [x] %r:%r" % (method.routing_key, body))
channel.basic_consume(callback,
queue=queue_name,
no_ack=True)
channel.start_consuming()
How to write a spring boot rabbitmq sender code for this? What are the necessary things to be declared there? Please help.

Spring Boot is Java application, first of all.
So, you need to make yourself familiar with that language.
And the choice is correct: you really can send to the RabbitMQ from Java and receive from the queue in any other client.
Spring Boot provides for you RabbitTemplate bean. So, if the story is about to send you just need to inject such a bean and use its API to send:
#Autowired
RabbitTemplate rabbitTemplate;
...
this.rabbitTemplate.convertAndSend("topic_logs", binding_key, data);
See Reference Manual for more info.

With spring integration, first you need to create a configuration class:
#Configuration
public class RabbitConfig {
#Autowired
private ConnectionFactory connectionFactory;
#Bean
DirectExchange dropfileExchange() {
return new DirectExchange("exchange_name", true, false);
}
#Bean
public Queue dropfileQueue() {
return new Queue("queue_name", true);
}
#Bean
Binding dropfileExchangeBinding(DirectExchange dropfileExchange, Queue dropfileQueue) {
return BindingBuilder.bind(dropfileQueue).to(dropfileExchange).with("key_name");
}
#Bean
public RabbitTemplate dropfileExchangeTemplate() {
RabbitTemplate rt = new RabbitTemplate(connectionFactory);
rt.setExchange("exchange_name");
rt.setRoutingKey("key_name");
rt.setConnectionFactory(connectionFactory);
return rt;
}
#Bean
public RabbitMessagingTemplate rabbitMessagingTemplate() {
return new RabbitMessagingTemplate(dropfileExchangeTemplate());
}
}
Then create a gateway service:
#MessagingGateway
public interface DropfileMessageGateway {
#Gateway(requestChannel = "channel_name")
void generate(String payload);
}
And then specify the producer flow using Java DSL as follows:
#Bean
public IntegrationFlow toOutboundQueueFlow() {
return IntegrationFlows.from("channel_name")
.transform(Transformers.toJson())
.handle(Amqp.outboundAdapter(rabbitConfig.dropfileExchangeTemplate())).get();
}
That will read a message from the channel, transform it to JSON and then dispatch it using an outbound adapter to the rabbit exchange.
Note that the messages go to the channel via the messaging gateway, so the channel name in both should be the same.
Do not forget to include the appropriate dependencies.

spring amqp-outbound gateway to produce reply from a different thead (Like jms-outbound gateway)

Problem statement:
Spring amqp-outbound gateway to produce reply from a different thread (Like jms-outbound gateway, having different queue, correlate the request/response using correlation key).
Unable to correlate the message with this example.
Spring integration
<int:gateway id="outboundGateway" service-interface="com.amqp.outbound.gateway.OutboundGateway"
default-reply-channel="defaultReplyChannel" >
<int:method name="process" request-channel="inboundRequestChannel"/>
</int:gateway>
<int:channel id="defaultReplyChannel"/>
<int:channel id="inboundRequestChannel"/>
<int:channel id="enrichedInboundRequestChannel"/>
<int:channel id="processAuthRequestChannel"/>
<int:channel id="postProcessorChannel"/>
<int:chain input-channel="inboundRequestChannel" output-channel="enrichedInboundRequestChannel">
<int:service-activator id="serviceActivator"
ref="ouboundService" method="createRequest"/>
</int:chain>
<int-amqp:outbound-gateway id="outboundGtwyId" header-mapper="headerMapper"
request-channel="enrichedInboundRequestChannel"
reply-channel="defaultReplyChannel"
amqp-template="template"
reply-timeout="30000"
exchange-name="request_exchange"
routing-key="request_exchange_queue"/>
<int-amqp:inbound-channel-adapter id="amqpMessageDriven" queue-names="request_queue"
connection-factory="rabbitConnectionFactory" channel="processAuthRequestChannel"/>
<int:service-activator id="serviceActivator"
ref="ouboundService" input-channel="processAuthRequestChannel" output-channel="postProcessorChannel"
method="processRequest"/>
<int-amqp:outbound-channel-adapter amqp-template="template" channel="postProcessorChannel"
header-mapper="headerMapper" exchange-name="reply_exchange" routing-key="reply_exchange_queue"/>
<bean id="headerMapper" class="org.springframework.integration.amqp.support.DefaultAmqpHeaderMapper"/>
Config
#Bean
public RabbitTemplate template(ConnectionFactory rabbitConnectionFactory){
final RabbitTemplate template = new RabbitTemplate(rabbitConnectionFactory);
template.setQueue("reply_queue");
return template;
}
#Bean
public Binding binding(){
return BindingBuilder.bind(this.queue()).to(this.exchange()).with("request_exchange_queue");
}
#Bean
public DirectExchange exchange(){
return new DirectExchange("request_exchange");
}
#Bean
public Queue queue(){
return new Queue("request_queue", true, false, true);
}
#Bean
public Binding bindingReply(){
return BindingBuilder.bind(this.queue()).to(this.exchange()).with("reply_exchange_queue");
}
#Bean
public DirectExchange exchangeReply(){
return new DirectExchange("reply_exchange");
}
#Bean
public Queue replyQueue(){
return new Queue("reply_queue", true, false, true);
}
Service
#Service
public final class OuboundService {
public Message createRequest(String message){
System.out.println("Inside createRequest : "+ message);
final String transactionId = UUID.randomUUID().toString();
final Message builtMessage = MessageBuilder.withBody(message.getBytes())
.setContentType(MessageProperties.CONTENT_TYPE_TEXT_PLAIN)
.setHeader(AmqpHeaders.CORRELATION_ID, transactionId)
.build();
return builtMessage;
}
public Message processRequest(Message message){
System.out.println("Inside process Request : "+ new String(message.getBody()));
System.out.println("Header values : "+message.getMessageProperties().getHeaders());
final Message result = MessageBuilder.withBody("Successful".getBytes()).copyProperties(message.getMessageProperties())
.copyHeaders(message.getMessageProperties().getHeaders()).build();
return result;
}
}
Error:
org.springframework.integration.handler.ReplyRequiredException: No reply produced by handler 'outboundGtwyId', and its 'requiresReply' property is set to true.
GitHub source code (Resolved Solution)
https://github.com/kingkongprab/spring-amqp-outbound-gateway

The correlation is done in the Spring AMQP as well. See its RabbitTemplate#sendAndRecevie() for more info. Also there is a good documentation on the matter in the Reference Manual.
Spring Integration with its AbstractAmqpOutboundEndpoint and AmqpInboundGateway implementations provides out-of-the-box request-reply correlation solution. If you are not able to use AmqpInboundGateway on the server side, you should ensure the correlationId transfer from received request to the reply to send back. Yes, you can use dedicated exchange for replies and that is what supported by the RabbitTemplate#setQueue() to wait for replies on the client, outbound side. But that still isn't going to work without proper correlation transferring. Also see https://docs.spring.io/spring-integration/docs/4.3.12.RELEASE/reference/html/amqp.html#amqp-message-headers for the info how headers (including correlationId) are mapped in Spring Integration.
UPDATE
Thank you for sharing your application.
Well, now I see several problems:
You are definitely missing the replyQueue binding:
#Bean
public Binding bindingReply(){
return BindingBuilder.bind(this.replyQueue()).to(this.exchangeReply()).with("reply_exchange_queue");
}
RabbitTemplate must use setReplyAddress(). You have to configure MessageListenerContainer for the reply_queue and have RabbitTemplate as a listener:
#Bean
public RabbitTemplate template(ConnectionFactory rabbitConnectionFactory){
final RabbitTemplate template = new RabbitTemplate(rabbitConnectionFactory);
template.setReplyAddress(replyQueue().getName());
return template;
}
#Bean
public MessageListenerContainer replyContainer(RabbitTemplate template) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(template.getConnectionFactory());
container.setQueues(replyQueue());
container.setMessageListener(template);
return container;
}
Your OuboundService with org.springframework.amqp.core.Message manipulation is useless. The Channel Adapters don't know about this type of payload and your custom Message just becomes as a serialized body of another org.springframework.amqp.core.Message. I have changed it to this and everything works well:
public String createRequest(String message){
System.out.println("Inside createRequest : "+ message);
return message;
}
public Message processRequest(Message message){
System.out.println("Inside process Request : " + message);
return message;
}
Anyway I suggest you to rethink your design and come back to the AmqpInboundGateway.
BTW in the final solution you don't need to care about any correlation. The Framework does that for you automatically.

Spring Integration - How to implement an asynchronous TCP Socket requests/responses over the same connection?

I have a Python TCP Socket server service which:
Allows only one client connection at time;
Its inputstream/outputstream operates independently.
On the other side, I have a Java Spring Boot client application using Spring Integration. My actual TCP Socket configurator
implementation uses:
#MessagingGateway(defaultRequestChannel = REQUEST_CHANNEL, errorChannel = ERROR_CHANNEL)
public interface ClientGtw {
Future<Response> send(Request request);
}
#Bean
#ServiceActivator(inputChannel = REQUEST_CHANNEL)
public MessageHandler outboundGateway(TcpNioClientConnectionFactory connectionFactory) {
TcpOutboundGateway gateway = new TcpOutboundGateway();
gateway.setConnectionFactory(connectionFactory);
gateway.setRequestTimeout(TimeUnit.SECONDS.toMillis(timeout));
gateway.setRemoteTimeout(TimeUnit.SECONDS.toMillis(timeout));
return gateway;
}
#Bean
public TcpNioClientConnectionFactory clientConnectionFactory(AppConfig config) {
Host host = getHost(config);
TcpNioClientConnectionFactory factory = new TcpNioClientConnectionFactory(host.name, host.port);
factory.setSingleUse(false);
factory.setSoTimeout((int) TimeUnit.SECONDS.toMillis(timeout));
SerializerDeserializer sd = new SerializerDeserializer();
factory.setDeserializer(sd);
factory.setSerializer(sd);
return factory;
}
This actual approach works fine, however, when a request is sent it hangs the connection until a response is received. This is a problem due the fact that some times a request can get too much time to receive a response and the system has other requests incomming whose response can be achieved faster. I would like to send and receive as much as possible requests and responses independetly (decoupled between them). The object transported (serialized and deserialized) contains a key pair that can do the correct correlation.
TL;DR: How to implement an Asynchronous requests/responses over the same connection?
The Spring TcpOutboundGateway javadoc mentions: Use a pair of outbound/inbound adapters for that use case.
So, in addition to the declaration above:
1st Attempt
#Bean
public TcpInboundGateway inboundGateway(AbstractServerConnectionFactory connectionFactory) {
TcpInboundGateway gateway = new TcpInboundGateway();
gateway.setConnectionFactory(connectionFactory);
gateway.setRequestTimeout(TimeUnit.SECONDS.toMillis(timeout));
return gateway;
}
#Bean
public AbstractServerConnectionFactory serverFactory(AppConfig config) {
Host host = getHost(config);
AbstractServerConnectionFactory connectionFactory = new TcpNetServerConnectionFactory(host.port);
connectionFactory.setSingleUse(true);
connectionFactory.setSoTimeout(timeout);
return connectionFactory;
}
The requests are blocked until a response is delivered as before.
2nd Attempt
#Bean
public TcpInboundGateway inboundGateway(TcpNioClientConnectionFactory connectionFactory) {
TcpInboundGateway gateway = new TcpInboundGateway();
gateway.setConnectionFactory(connectionFactory);
gateway.setRequestTimeout(TimeUnit.SECONDS.toMillis(timeout));
gateway.setClientMode(true);
return gateway;
}
org.springframework.integration.ip.tcp.connection.TcpNioClientConnectionFactory may only be used by one inbound adapter
Any clue?

Use a pair of channel adapters instead of an outbound gateway. Instead of using a MessagingGateway, you can do the correlation yourself in your application, or you can use the same technique as is used in the tcp-client-server-multiplex sample app. It uses an aggregator to aggregate a copy of the outbound message with an inbound message, replying to the gateway.
It's old, and uses XML configuration, but the same techniques apply.
<publish-subscribe-channel id="input" />
<ip:tcp-outbound-channel-adapter id="outAdapter.client"
order="2"
channel="input"
connection-factory="client" /> <!-- Collaborator -->
<!-- Also send a copy to the custom aggregator for correlation and
so this message's replyChannel will be transferred to the
aggregated message.
The order ensures this gets to the aggregator first -->
<bridge input-channel="input" output-channel="toAggregator.client"
order="1"/>
<!-- Asynch receive reply -->
<ip:tcp-inbound-channel-adapter id="inAdapter.client"
channel="toAggregator.client"
connection-factory="client" /> <!-- Collaborator -->
<!-- dataType attribute invokes the conversion service, if necessary -->
<channel id="toAggregator.client" datatype="java.lang.String" />
<aggregator input-channel="toAggregator.client"
output-channel="toTransformer.client"
expire-groups-upon-completion="true"
expire-groups-upon-timeout="true"
discard-channel="noResponseChannel"
group-timeout="1000"
correlation-strategy-expression="payload.substring(0,3)"
release-strategy-expression="size() == 2" />
<channel id="noResponseChannel" />
<service-activator input-channel="noResponseChannel" ref="echoService" method="noResponse" />
<transformer input-channel="toTransformer.client"
expression="payload.get(1)"/> <!-- The response is always second -->
(This simple sample correlates on the first 3 bytes).

Gary, thanks for your guidance.
To solve this issue is important to first understand Messaging Channel types.
So, in the configurer class:
#Bean(name = REQUEST_CHANNEL)
public DirectChannel sender() {
return new DirectChannel();
}
#Bean(name = RESPONSE_CHANNEL)
public PollableChannel receiver() {
return new QueueChannel();
}
#Bean
#ServiceActivator(inputChannel = REQUEST_CHANNEL)
public TcpSendingMessageHandler outboundClient(TcpNioClientConnectionFactory connectionFactory) {
TcpSendingMessageHandler outbound = new TcpSendingMessageHandler();
outbound.setConnectionFactory(connectionFactory);
outbound.setRetryInterval(TimeUnit.SECONDS.toMillis(timeout));
outbound.setClientMode(true);
return outbound;
}
#Bean
public TcpReceivingChannelAdapter inboundClient(TcpNioClientConnectionFactory connectionFactory) {
TcpReceivingChannelAdapter inbound = new TcpReceivingChannelAdapter();
inbound.setConnectionFactory(connectionFactory);
inbound.setRetryInterval(TimeUnit.SECONDS.toMillis(timeout));
inbound.setOutputChannel(receiver());
inbound.setClientMode(true);
return inbound;
}
This scratch #Singleton class illustrates how to operate the requests and responses (considering that requests and responses contains a UID to correlate them):
#Autowired
private DirectChannel sender;
#Autowired
private PollableChannel receiver;
private BlockingQueue<Request> requestPool = new LinkedBlockingQueue<>();
private Map<String, Response> responsePool = Collections.synchronizedMap(new HashMap<>());
#PostConstruct
private void init() {
new Receiver().start();
new Sender().start();
}
/*
* It can be called as many as necessary without hanging for a response
*/
public void send(Request req) {
requestPool.add(req);
}
/*
* Check for a response until a socket timout
*/
public Response receive(String key) {
Response res = responsePool.get(key);
if (res != null) {
responsePool.remove(key);
}
return res;
}
private class Receiver extends Thread {
#Override
public void run() {
while (true) {
try {
tcpReceive();
Thread.sleep(250);
} catch (InterruptedException e) { }
}
}
private void tcpReceive() {
Response res = (Message<Response>) receiver.receive();
if (res != null) {
responsePool.put(res.getUID(), res);
}
}
}
private class Sender extends Thread {
#Override
public void run() {
while (true) {
try {
tcpSend();
Thread.sleep(250);
} catch (InterruptedException e) { }
}
}
private void tcpSend() {
Request req = requestPool.poll(125, TimeUnit.MILLISECONDS);
if (req != null) {
sender.send(MessageBuilder.withPayload(req).build());
}
}
}
UPDATED
I forgot to mention this:
#Bean
public TcpNioClientConnectionFactory clientConnectionFactory(Config config) {
// Get host properties
Host host = getHost(config);
// Create socket factory
TcpNioClientConnectionFactory factory = new TcpNioClientConnectionFactory(host.name, host.port);
factory.setSingleUse(false); // IMPORTANT FOR SINGLE CHANNEL
factory.setSoTimeout((int) TimeUnit.SECONDS.toMillis(timeout));
return factory;
}
Feel free to make any considerations.