I have the Rabbit MQ broker for communicating asynchronously between services. Service A is sending messages to the queue. I checked the queue and the messages from Service A have arrived:
I am trying to create a listener in the Service B in order to consume the messages produced by Service A. I verified like below to check if Service B is connected with RabbitMQ and it seems to be connected successfully.
The problem is that Service B started successfully but it is receiving messages from Rabbit MQ.
Below is the implementation of the listener:
#Slf4j
#Component
public class EventListener {
public static final String QUEUE_NAME = "events";
#RabbitListener(
bindings = {
#QueueBinding(
value = #Queue(QUEUE_NAME),
exchange = #Exchange("exchange")
)
}
)
public void handleTaskPayload(#Payload String payload) {
System.out.println(payload);
}
}
I verified the queue and exchange information in the Rabbit MQ and they are correct.
Everything is working correctly and there is no error thrown in service A or service B which makes this problem much harder to debug.
I tried to retrieve the message from the queue getMessage of RabbitMQ the message is like the below:
{"id":"1",:"name:"Test","created":null}
I will appreciate any help or guidance towards the solution of this problem.
Best Regards,
Rando.
P.S
I created a new test queue like the below and published some messages:
Modified the listener code like below and still wasn't able to trigger listener to listen to the queue events:
#Slf4j
#Component
public class RobotRunEventListener {
public static final String QUEUE_NAME = "test";
#RabbitListener(
bindings = {
#QueueBinding(
value = #Queue(QUEUE_NAME),
key = "test",
exchange = #Exchange("default")
)
}
)
public void handleTaskPayload(#Payload String payload) {
System.out.println(payload);
}
Try this approach:
#RabbitListener(queues = "test")
public void receive(String in, #Headers Map<String, Object> headers) throws IOException {
}
The problem was that the spring boot app that I was working on had a #Conditional(Config.class) that prevented the creation of the bean below:
#Slf4j
#Conditional(Config.class)
#EnableRabbit
public class InternalRabbitBootstrapConfiguration {
#Bean
public RabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setMaxConcurrentConsumers(5);
return factory;
}
...
which resulted in the spring boot app not listening to Rabbit MQ events. The Config.class required a specific profile in order to enable the app to listen to Rabbit MQ events.
public class DexiModeCondition implements Condition {
#Override
public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {
String[] activeProfiles = context.getEnvironment().getActiveProfiles();
return activeProfiles[0].equalsIgnoreCase(mode);
}
}
Related
I have some Spring applications that communicate between them using RabbitMQ as broker. I can send and receive messages asynchronously between them. But now, I need one application to send a message to another one and wait for the response. So, for this I am trying to implement the RPC pattern. It is working, but the problem is that I could only do it using temporary queues generated by Spring.
https://www.rabbitmq.com/tutorials/tutorial-six-spring-amqp.html
This is the code that sends the message and wait for the response.
public void send() {
....
Integer response = (Integer) template.convertSendAndReceive(exchange.getName(), "rpc", "message");
...
}
When I send the message, the execution is blocked until the response is received and a temporary queue is created by Spring for the response, as expected.
But what I need is to use a specific and fixed queue, defined by me, to receive the responses. I need responses to be sent to an exchange with a routing key pointing to the fixed response queue (doing this I'll be able to send the responses to another queue too, that will be logging all responses).
I tried setting the "setReplyTo" property to the message, but is not working.
What version are you using? With modern versions, direct reply_to is used by default, but you can revert to using a temporary queue by setting a property on the template.
https://docs.spring.io/spring-amqp/docs/current/reference/html/#direct-reply-to
To use a named reply queue, see the documentation about how to set up a reply container, with the template as the message listener:
https://docs.spring.io/spring-amqp/docs/current/reference/html/#message-correlation-with-a-reply-queue
and
https://docs.spring.io/spring-amqp/docs/current/reference/html/#reply-listener
EDIT
The template will block until the corresponding reply is passed into it by the reply container (or it times out).
#SpringBootApplication
public class So68986604Application {
public static void main(String[] args) {
SpringApplication.run(So68986604Application.class, args);
}
#RabbitListener(queues = "foo")
public String listen(String in) {
System.out.println(in);
return in.toUpperCase();
}
#Bean
Queue foo() {
return new Queue("foo");
}
#Bean
Queue replies() {
return new Queue("foo.replies");
}
#Bean
SimpleMessageListenerContainer replyContainer(ConnectionFactory cf, RabbitTemplate template) {
SimpleMessageListenerContainer replyer = new SimpleMessageListenerContainer(cf);
replyer.setQueueNames("foo.replies");
replyer.setMessageListener(template);
template.setReplyAddress("foo.replies");
return replyer;
}
#Bean
public ApplicationRunner runner(RabbitTemplate template) {
return args -> {
System.out.println(template.convertSendAndReceive("", "foo", "test"));
};
}
}
test
TEST
In my use case I need to do request-reply call to a remote system via managed queues. Using Spring Boot and IBM's MQ starter I have the problem that the application wants to create dynamic/temporary reply queues instead of using the already existing managed queue.
Configuration is set up here
#EnableJms
#Configuration
public class QueueConfiguration {
#Bean
public MQQueueConnectionFactory connectionFactory() throws JMSException {
MQQueueConnectionFactory factory = new MQQueueConnectionFactory();
factory.setTransportType(CT_WMQ); // is 1
factory.setHostName(queueProperties.getHost());
factory.setPort(queueProperties.getPort());
factory.setChannel(queueProperties.getChannel()); // combo of ${queueManager}%${channel}
return factory;
}
#Bean
public JmsMessagingTemplate messagingTemplate(ConnectionFactory connectionFactory) {
JmsMessagingTemplate jmt = new JmsMessagingTemplate(connectionFactory);
jmt.setDefaultDestinationName(queueProperties.getQueueName());
return jmt;
}
#Bean
public Jaxb2Marshaller marshaller() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setPackagesToScan("com.foo.model");
return marshaller;
}
#Bean
public MessageConverter messageConverter(Jaxb2Marshaller marshaller) {
MarshallingMessageConverter converter = new MarshallingMessageConverter();
converter.setMarshaller(marshaller);
converter.setUnmarshaller(marshaller);
return converter;
}
}
Usage is pretty straight forward: Take the object convert and send it. Wait for response receive
and convert it.
#Component
public class ExampleSenderReceiver {
#Autowired
private JmsMessagingTemplate jmsMessagingTemplate;
#Override
#SneakyThrows
public ResponseExample sendAndReceive(RequestExample request, String correlationId) {
MessagePostProcessor mpp = message -> {
message = MessageBuilder.fromMessage(message)
.setHeader(JmsHeaders.CORRELATION_ID, correlationId)
// .setHeader(JmsHeaders.REPLY_TO, "DEV.QUEUE.3") this triggers queue creation
.build();
return message;
};
String destination = Objects.requireNonNull(jmsMessagingTemplate.getDefaultDestinationName());
return jmsMessagingTemplate.convertSendAndReceive(destination, request, ResponseExample.class, mpp);
}
I read already a lot of IBM documentation and think, I need to set the message type to "MQMT_REQUEST" but I do not find the right spot to do so.
Update
Added Spring Integration as Gary proposed and added a configuration for JmsOutboundGateway
#Bean
public MessageChannel requestChannel() {
return new DirectChannel();
}
#Bean
public QueueChannel responseChannel() {
return new QueueChannel();
}
#Bean
#ServiceActivator(inputChannel = "requestChannel" )
public JmsOutboundGateway jmsOutboundGateway( ConnectionFactory connectionFactory) {
JmsOutboundGateway gateway = new JmsOutboundGateway();
gateway.setConnectionFactory(connectionFactory);
gateway.setRequestDestinationName("REQUEST");
gateway.setReplyDestinationName("RESPONSE");
gateway.setReplyChannel(responseChannel());
gateway.setCorrelationKey("JMSCorrelationID*");
gateway.setIdleReplyContainerTimeout(2, TimeUnit.SECONDS);
return gateway;
}
And adapted my ExampleSenderReceiver class
#Autowired
#Qualifier("requestChannel")
private MessageChannel requestChannel;
#Autowired
#Qualifier("responseChannel")
private QueueChannel responseChannel;
#Override
#SneakyThrows
public ResponseExample sendAndReceive(RequestExample request, String correlationId) {
String xmlContent = "the marshalled request object";
Map<String, Object> header = new HashMap<>();
header.put(JmsHeaders.CORRELATION_ID, correlationId);
GenericMessage<String> message1 = new GenericMessage<>(xmlContent, header);
requestChannel.send(message1);
log.info("send done" );
Message<?> receive = responseChannel.receive(1500);
if(null != receive){
log.info("incoming: {}", receive.toString());
}
}
The important part is gateway.setCorrelationKey("JMSCorrelationID*");
Without that line the correlationId was not propagated correct.
Next step is re-adding MessageConverters and make it nice again.
Thank you.
The default JmsTemplate (used by the JmsMessagingTemplate) always uses a temporary reply queue. You can subclass it and override doSendAndReceive(Session session, Destination destination, MessageCreator messageCreator) to use your managed queue instead.
However, it will only work if you have one request outstanding at a time (e.g. all run on a single thread). You will also have to add code for discarding "late" arrivals by checking the correlation id.
You can use async sends instead and handle replies on a listener container and correlate the replies to the requests.
Consider using spring-integration-jms and its outbound gateway instead - it has much more flexibility in reply queue handling (and does all the correlation for you).
https://docs.spring.io/spring-integration/reference/html/jms.html#jms-outbound-gateway
You are missing the queue manager.
ibm:
mq:
queueManager: QM1
channel: chanel
connName: localhost(1414)
user: admin
password: admin
I'm trying to receive emails from gmail server and then process them somehow. So every time when new email arrives to the mail server, my application should download it and process it, it means call other services asynchronously which will be registered as listeners.
I'm quite new in spring integration and I don't fully understand how does it work or if it is correct.
1) So far, I have this code -- I'm able to read all emails but I'm not sure if they are processed asynchronously or not or if it is a correct way?
ListeningExample class
#Configuration
public class ListeningExample {
#Bean
public HeaderMapper<MimeMessage> mailHeaderMapper() {
return new DefaultMailHeaderMapper();
}
#Bean
public IntegrationFlow imapMailFlow() {
IntegrationFlow flow = IntegrationFlows
.from(Mail.imapInboundAdapter("imap://user:pwd#imap.gmail.com/INBOX")
.userFlag("testSIUserFlag")
.javaMailProperties(javaMailProperties()),
e -> e.autoStartup(true)
.poller(p -> p.fixedDelay(5000)))dostane to detailni zpravy
.transform(Mail.toStringTransformer())
.channel(MessageChannels.queue("imapChannel"))
.get();
return flow;
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
pollerMetadata.setTrigger(new PeriodicTrigger(1000));
return pollerMetadata;
}
}
MailRecieverService class
#Service
public class MailRecieverService {
private List<EmailAction> services;
#Bean
#ServiceActivator(inputChannel = "imapChannel")
public MessageHandler processNewEmail() {
MessageHandler messageHandler = new MessageHandler() {
#Override
public void handleMessage(org.springframework.messaging.Message<?> message) throws MessagingException {
System.out.println("New email:" + message.toString());
//Processing emails do with them something..
for (EmailAction emailAction : services) {
emailAction.performAction(null);
}
}
};
return messageHandler;
}
}
Main class
#SpringBootApplication
#EnableIntegration
public class Main extends SpringBootServletInitializer {
#Override
protected SpringApplicationBuilder configure(SpringApplicationBuilder application) {
return application.sources(Main.class);
}
public static void main(String[] args) throws Exception {
SpringApplicationBuilder builder = new SpringApplicationBuilder(Main.class);
builder.headless(false).run(args);
}
}
2) Is there also a possibility in spring boot to move/delete emails, create folders, and do other actions with an email account or do I have to use javax.mail library on my own?
If yes, could you provide some examples?
Your integration flow is correct. The IMAP Inbound Channel Adapter is async by its nature and it spawns a thread to loop new emails. However if you worry about your for (EmailAction emailAction : services) {, then you need to think making this part as async by yourself. You definitely can use a PublishSubscribeChannel with an Executor and several subscribers for those services to really process the same email message in those services in parallel.
All the other stuff about delete, create and other email management operations are out of Enterprise Integration Patterns scope. Therefore there is no any high level Spring API to do those operation from the application. Consider to use Java Mail API directly: https://javaee.github.io/javamail/#Project_Documentation
According to the documentation, ImapIdleChannelAdapter is asynchronous.
https://docs.spring.io/spring-integration/docs/current/reference/html/mail.html#mail-inbound
To do an action after the processing of the message has finished you can wrap it in a transaction, and perform an action after commit:
In the example of the documentation it is moving the email to another folder.
https://docs.spring.io/spring-integration/docs/current/reference/html/mail.html#mail-tx-sync
In an effort to learn Apache Kafka, I’ve developed a Spring Boot application that sends messages to a Kafka topic if I send a POST request to a controller that calls a KafkaTemplate send method. I’m running Ubuntu 19.04 and successfully set up and installed Kafka and Zookeeper locally. Everything works fine.
The problem happens when I shut down either Zookeeper or Kafka. If I do this then on startup the Kafka AdminClient of my application periodically tries to find a broer but sends this message to the console
Connection to node -1 could not be established. Broker may not be available.
I implemented the fixes suggested here Kafka + Zookeeper: Connection to node -1 could not be established. Broker may not be available and here Spring-Boot and Kafka : How to handle broker not available?. But if I run a maven clean install then the build never finishes if Zookeeper and Kafka aren’t running. Why is this and is there a way to configure the application so that it checks for Kafka availability on startup and gracefully handles when the service is unavailable?
Here is my service class that calls the KafkaTemplate
#Autowired
public PingMessageServiceImpl(KafkaTemplate kafkaTemplate, KafkaTopicConfiguration kafkaTopicConfiguration) {
this.kafkaTemplate = kafkaTemplate;
this.kafkaTopicConfiguration = kafkaTopicConfiguration;
}
#Override
public void sendMessage(String message) {
log.info(String.format("Received following ping message %s", message));
if (!isValidPingRequest(message)) {
log.warn("Received invalid ping request");
throw new InvalidPingRequestException();
}
log.info(String.format("Sending message=[%s]", message));
ListenableFuture<SendResult<String, String>> future = kafkaTemplate.send(kafkaTopicConfiguration.getPingTopic(), message);
future.addCallback(buildListenableFutureCallback(message));
}
private boolean isValidPingRequest(String message) {
return "ping".equalsIgnoreCase(message);
}
private ListenableFutureCallback<SendResult<String, String>> buildListenableFutureCallback(String message) {
return new ListenableFutureCallback<SendResult<String, String>>() {
#Override
public void onSuccess(SendResult<String, String> result) {
log.info(String.format("Sent message=[%s] with offset=[%d]", message, result.getRecordMetadata().offset()));
}
#Override
public void onFailure(Throwable ex) {
log.info(String.format("Unable to send message=[%s] due to %s", message, ex.getMessage()));
}
};
}
Here is the configuration class that I use to extract configuration properties for Kafka from the properties file
#NotNull(message = "bootstrapAddress cannot be null")
#NotBlank(message = "bootstrapAddress cannot be blank")
private String bootstrapAddress;
#NotNull(message = "pingTopic cannot be null")
#NotBlank(message = "pingTopic cannot be blank")
private String pingTopic;
#NotNull(message = "reconnectBackoffMs cannot be null")
#NotBlank(message = "reconnectBackoffMs cannot be blank")
#Value("${kafka.reconnect.backoff.ms}")
private String reconnectBackoffMs;
#Bean
public KafkaAdmin kafkaAdmin() {
Map<String, Object> configurations = new HashMap<>();
configurations.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
configurations.put(AdminClientConfig.RECONNECT_BACKOFF_MS_CONFIG, reconnectBackoffMs);
return new KafkaAdmin(configurations);
}
#Bean
public NewTopic pingTopic() {
return new NewTopic(pingTopic, 1, (short) 1);
}
#PostConstruct
private void displayOnStartup() {
log.info(String.format("bootstrapAddress is %s", bootstrapAddress));
log.info(String.format("reconnectBackoffMs is %s", reconnectBackoffMs));
}
If you have any Spring-boot integration test while loading the ApplicationContext spring kafka beans like KafakTemplate,KafkaAdmin will try to connect the kafka server with the properties specified in yml or properties file
So to avoid this you can use spring-embedded-kafka-server, so that kafka beans will connect to embedded server during test execution.
Or simple you can just mock the KafakTemplate and KafkaAdmin beans using #MockBean annotation in the integration test cases
I want to write an application wherein I need a producer sending messages using spring boot rabbitmq and the receiver is written in python. The receiver part was easy-
receive.py
#!/usr/bin/env python
import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.exchange_declare(exchange='topic_logs',
exchange_type='topic')
result = channel.queue_declare(exclusive=True)
queue_name = result.method.queue
binding_keys = sys.argv[1:]
if not binding_keys:
sys.stderr.write("Usage: %s [binding_key]...\n" % sys.argv[0])
sys.exit(1)
for binding_key in binding_keys:
channel.queue_bind(exchange='topic_logs',
queue=queue_name,
routing_key=binding_key)
print(' [*] Waiting for logs. To exit press CTRL+C')
def callback(ch, method, properties, body):
print(" [x] %r:%r" % (method.routing_key, body))
channel.basic_consume(callback,
queue=queue_name,
no_ack=True)
channel.start_consuming()
How to write a spring boot rabbitmq sender code for this? What are the necessary things to be declared there? Please help.
Spring Boot is Java application, first of all.
So, you need to make yourself familiar with that language.
And the choice is correct: you really can send to the RabbitMQ from Java and receive from the queue in any other client.
Spring Boot provides for you RabbitTemplate bean. So, if the story is about to send you just need to inject such a bean and use its API to send:
#Autowired
RabbitTemplate rabbitTemplate;
...
this.rabbitTemplate.convertAndSend("topic_logs", binding_key, data);
See Reference Manual for more info.
With spring integration, first you need to create a configuration class:
#Configuration
public class RabbitConfig {
#Autowired
private ConnectionFactory connectionFactory;
#Bean
DirectExchange dropfileExchange() {
return new DirectExchange("exchange_name", true, false);
}
#Bean
public Queue dropfileQueue() {
return new Queue("queue_name", true);
}
#Bean
Binding dropfileExchangeBinding(DirectExchange dropfileExchange, Queue dropfileQueue) {
return BindingBuilder.bind(dropfileQueue).to(dropfileExchange).with("key_name");
}
#Bean
public RabbitTemplate dropfileExchangeTemplate() {
RabbitTemplate rt = new RabbitTemplate(connectionFactory);
rt.setExchange("exchange_name");
rt.setRoutingKey("key_name");
rt.setConnectionFactory(connectionFactory);
return rt;
}
#Bean
public RabbitMessagingTemplate rabbitMessagingTemplate() {
return new RabbitMessagingTemplate(dropfileExchangeTemplate());
}
}
Then create a gateway service:
#MessagingGateway
public interface DropfileMessageGateway {
#Gateway(requestChannel = "channel_name")
void generate(String payload);
}
And then specify the producer flow using Java DSL as follows:
#Bean
public IntegrationFlow toOutboundQueueFlow() {
return IntegrationFlows.from("channel_name")
.transform(Transformers.toJson())
.handle(Amqp.outboundAdapter(rabbitConfig.dropfileExchangeTemplate())).get();
}
That will read a message from the channel, transform it to JSON and then dispatch it using an outbound adapter to the rabbit exchange.
Note that the messages go to the channel via the messaging gateway, so the channel name in both should be the same.
Do not forget to include the appropriate dependencies.