Develop Reference

Spring Integration: delete (sftp) remote file after it has been persisted to database

(for context, please see my previous question)
Still struggling to get a file from an SFTP server, persist it's content to a database and remove that file afterwards, if it was persisted without errors. I just can get the handling of handlers, gateways and flows right.
I need guidanc, please
What I have:
#Configuration
#EnableIntegration
class Sftp2DB {
#Bean
#InboundChannelAdapter(channel = "transform")
public MessageSource<InputStream> source() {
return Sftp
.inboundStreamingAdapter(template(this.sessionFactory))
.remoteDirectory("inbound")
.get();
}
#Transformer(inputChannel="transform", outputChannel = "persist")
public Message<MyEntity> transform(final Message<InputStream> in) throws IOException {
var entity = new MyEntity();
entity.setContent(in.getPayload().readAllBytes());
entity.setFilename(in.getHeaders().get(FileHeaders.FILENAME, String.class));
return MessageBuilder.withPayload(entity).build();
}
#ServiceActivator(inputChannel = "persist", outputChannel = "remove")
public JpaOutboundGateway persist() {
return Jpa
.updatingGateway(this.entityManager)
.entityClass(MyEntity.class)
.persistMode(PersistMode.PERSIST)
.get();
}
#ServiceActivator(inputChannel = "remove")
public AbstractRemoteFileOutboundGateway<LsEntry> remove() {
return Sftp
.outboundGateway(
this.sessionFactory,
"rm",
String.format("header['%s'] + '/' + header['%s']", FileHeaders.REMOTE_DIRECTORY, FileHeaders.REMOTE_FILE)
)
.get();
}
}
What I get:
2022-11-24 12:50:13.815 ERROR 948 --- [ scheduling-1] o.s.integration.handler.LoggingHandler : org.springframework.messaging.MessageHandlingException: error occurred in message handler [ServiceActivator for [org.springframework.integration.handler.MethodInvokingMessageProcessor#3be14a03] (Sftp2DB.remove.serviceActivator)]; nested exception is org.springframework.messaging.core.DestinationResolutionException: no output-channel or replyChannel header available, failedMessage=GenericMessage [payload=org.springframework.integration.jpa.outbound.JpaOutboundGateway#6a0e79fb, headers={id=788f63b5-ad62-de6b-bbb1-ecde94d23576, timestamp=1669290613815}]

There are two types of #ServiceActivator (and #Transformer etc).
POJO methods (like your transformer) and beans that define message handlers.
Your service activators need to be defined as #Bean s (like you did with your inbound channel adapter).
See https://docs.spring.io/spring-integration/docs/current/reference/html/configuration.html#annotations and https://docs.spring.io/spring-integration/docs/current/reference/html/configuration.html#annotations_on_beans

Spring Boot App not consuming queue messages

I have the Rabbit MQ broker for communicating asynchronously between services. Service A is sending messages to the queue. I checked the queue and the messages from Service A have arrived:
I am trying to create a listener in the Service B in order to consume the messages produced by Service A. I verified like below to check if Service B is connected with RabbitMQ and it seems to be connected successfully.
The problem is that Service B started successfully but it is receiving messages from Rabbit MQ.
Below is the implementation of the listener:
#Slf4j
#Component
public class EventListener {
public static final String QUEUE_NAME = "events";
#RabbitListener(
bindings = {
#QueueBinding(
value = #Queue(QUEUE_NAME),
exchange = #Exchange("exchange")
)
}
)
public void handleTaskPayload(#Payload String payload) {
System.out.println(payload);
}
}
I verified the queue and exchange information in the Rabbit MQ and they are correct.
Everything is working correctly and there is no error thrown in service A or service B which makes this problem much harder to debug.
I tried to retrieve the message from the queue getMessage of RabbitMQ the message is like the below:
{"id":"1",:"name:"Test","created":null}
I will appreciate any help or guidance towards the solution of this problem.
Best Regards,
Rando.
P.S
I created a new test queue like the below and published some messages:
Modified the listener code like below and still wasn't able to trigger listener to listen to the queue events:
#Slf4j
#Component
public class RobotRunEventListener {
public static final String QUEUE_NAME = "test";
#RabbitListener(
bindings = {
#QueueBinding(
value = #Queue(QUEUE_NAME),
key = "test",
exchange = #Exchange("default")
)
}
)
public void handleTaskPayload(#Payload String payload) {
System.out.println(payload);
}

Try this approach:
#RabbitListener(queues = "test")
public void receive(String in, #Headers Map<String, Object> headers) throws IOException {
}

The problem was that the spring boot app that I was working on had a #Conditional(Config.class) that prevented the creation of the bean below:
#Slf4j
#Conditional(Config.class)
#EnableRabbit
public class InternalRabbitBootstrapConfiguration {
#Bean
public RabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setMaxConcurrentConsumers(5);
return factory;
}
...
which resulted in the spring boot app not listening to Rabbit MQ events. The Config.class required a specific profile in order to enable the app to listen to Rabbit MQ events.
public class DexiModeCondition implements Condition {
#Override
public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {
String[] activeProfiles = context.getEnvironment().getActiveProfiles();
return activeProfiles[0].equalsIgnoreCase(mode);
}
}

SQS Message visibility timeout being set to 0 when exception is thrown and #JMSListener

I have a simple Spring Boot service that listens to an AWS SQS queue using JMSTemplate. Everything works as expected when the message is properly handled.
I am using CLIENT_ACKNOWLEDGE so when an exception is thrown during processing, the message is received again. However the Default Visibility Timeout setting on the SQS queue is being ignored and the message is being received again immediately.
The SQS queue is configured with a 30 second Default Visibility Timeout and a re-drive policy of 20 receives before putting the message on a DLQ.
I have disabled the service and used the SQS Console to verify that the Default Visibility Timeout is properly set. I have also tried adding the JMS Message to the method signature and performing manual validation.
Here is code for the JMS Configuration:
#Configuration
#EnableJms
class JmsConfig
{
#Bean
#Conditional(AWSEnvironmentCondition.class)
public SQSConnectionFactory connectionFactory(#Value("${AWS_REGION}") String awsRegion)
{
return new SQSConnectionFactory(
new ProviderConfiguration(),
AmazonSQSClientBuilder.standard()
.withRegion(Regions.fromName(awsRegion))
.withCredentials(new DefaultAWSCredentialsProviderChain())
);
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(ConnectionFactory connectionFactory)
{
DefaultJmsListenerContainerFactory factory =
new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setDestinationResolver(new DynamicDestinationResolver());
factory.setConcurrency("3-10");
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
factory.setErrorHandler(defaultErrorHandler());
return factory;
}
#Bean
public ErrorHandler defaultErrorHandler()
{
return new ErrorHandler()
{
#Override
public void handleError(Throwable throwable)
{
LOG.error("JMS message listener error: {}", throwable.getMessage());
}
};
}
#Bean
public JmsTemplate defaultJmsTemplate(ConnectionFactory connectionFactory)
{
return new JmsTemplate(connectionFactory);
}
}
And here is code for the Listener:
#Component
public class MessagingListener
{
#Autowired
private MessageService _messageService;
#Autowired
private Validator _validator;
#JmsListener(destination = "myqueue")
public void receiveMessage(String messageJson)
{
try
{
LOG.info("Received message");
// The following line throws an IOException is the message is not JSON.
MyMessage myMessage = MAPPER.readvalue(messageJson, MyMessage.class);
Set<ConstraintViolation<MyMessage>> _validator.validate(myMessage);
if (CollectionUtils.isNotEmpty(violations))
{
String errorMessage = violations.stream()
.map(v -> String.join(" : ", v.getPropertyPath().iterator().next().getName(),
v.getMessage()))
LOG.error("Exception occurred while validating the model, details: {}", errorMessage)
throw new ValidationException(errorMessage);
}
}
catch (IOException e)
{
LOG.error("Error parsing message", e);
throw new ValidationException("Error parsing message, details: " + e.getMessage());
}
}
}
When a message is placed on the SQS queue with either invalid JSON or JSON that that does not pass validation, the message is received 20 times very quickly and then ends up on the DLQ. What needs to be done so that the Default Visibility Timeout setting in SQS is respected?

In case of an exception, visibility timeout of the failed message is set to 0 via ChangeMessageVisibility so SQS will send this message immediately even though queue has a different visibilityTimeout setting.
How does that happen?
As you can see here, Spring JMS' AbstractMessageListenerContainer briefly does this:
try {
invokeListener(session, message); // This is your #JMSListener method
}
catch (JMSException | RuntimeException | Error ex) {
rollbackOnExceptionIfNecessary(session, ex);
throw ex;
}
commitIfNecessary(session, message);
On rollbackOnExceptionIfNecessary method, session.recover() will be invoked because:
session.getTransacted() will always be false since SQS does not support transactions. See here.
isClientAcknowledge(session) will return true because you're using CLIENT_ACKNOWLEDGE mode.
And lastly recover() of SQSSession negative acknowledges the message, which means setting visibilityTimeout of that specific message to 0, causes SQS to try sending that message immediately.
The easiest way to override this behavior would be implementing a CustomJmsListenerContainerFactory & CustomMessageListenerContainer instead of using DefaultJmsListenerContainerFactory & DefaultMessageListenerContainer.
public class CustomMessageListenerContainer extends DefaultMessageListenerContainer {
public CustomMessageListenerContainer() {
super();
}
#Override
protected void rollbackOnExceptionIfNecessary() {
// do nothing, so that "visibilityTimeout" will stay same
}
}
public class CustomJmsListenerContainerFactory {
#Override
protected DefaultMessageListenerContainer createContainerInstance() {
return new CustomMesageListenerContainer();
}
}
And make it a Spring bean either with #Component or just like you did in JmsConfig:
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(ConnectionFactory connectionFactory) {
DefaultJmsListenerContainerFactory factory = new CustomJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
// and set other stuff on factory
return factory;
}
NOTE:
If your application is consuming other type of data sources along SQS with JMS, make sure to use different Container and ContainerFactory for them so that rollbackOnExceptionIfNecessary behaves as expected.

How to listen to an existing queue in Spring AMQP?

I have a remote RabbitMQ server which has some queues I want to listen to. I tried this:
#RabbitListener(queues = "queueName")
public void receive(String message) {
System.out.println(message);
}
But it tried to create a new queue. Result is predictable - access denied.
o.s.a.r.listener.BlockingQueueConsumer : Failed to declare queue: queueName
I didn't declare any queue in any other way.
How can I listen to an existing queue on a remote server? Also, is there a way to check if this queue exists? And I saw this line
#RabbitListener(queues = "#{autoDeleteQueue2.name}")
in a tutorial. What does #{queueName.name} mean?
Logs and the beginning of the stack trace:
2018-08-30 22:10:21.968 WARN 12124 --- [cTaskExecutor-1] o.s.a.r.listener.BlockingQueueConsumer : Failed to declare queue: queueName
2018-08-30 22:10:21.991 WARN 12124 --- [cTaskExecutor-1] o.s.a.r.listener.BlockingQueueConsumer : Queue declaration failed; retries left=3
org.springframework.amqp.rabbit.listener.BlockingQueueConsumer$DeclarationException: Failed to declare queue(s):[queueName]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:711) ~[spring-rabbit-2.0.5.RELEASE.jar:2.0.5.RELEASE]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:588) ~[spring-rabbit-2.0.5.RELEASE.jar:2.0.5.RELEASE]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:996) [spring-rabbit-2.0.5.RELEASE.jar:2.0.5.RELEASE]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

Even if you don't have configuration permission on the broker, the queueDeclarePassive used by the listener is allowed (it checks for the presence of the queue).
o.s.a.r.listener.BlockingQueueConsumer : Failed to declare queue: queueName
That just means that the queue doesn't exist.
#RabbitListener(queues = "#{autoDeleteQueue2.name}")
That is used to get the queue name at runtime (when you have permission to create queues).
e.g.
#Bean
public AnonymousQueue autoDeleteQueue2() {
return new AnonymousQueue();
}
Spring will add that queue to the broker with a random, unique name. The listener is then configured with the actual queue name.

Here is an example on how to listen to a queue with rabbitMq :
#Component
public class RabbitConsumer implements MessageListener {
#RabbitListener(bindings =
#QueueBinding(
value = #Queue(value = "${queue.topic}", durable = "true"),
exchange = #Exchange(value = "${queue.exchange}", type = ExchangeTypes.FANOUT, durable = "true")
)
)
#Override
public void onMessage(Message message) {
// ...
}
}
And the config (application.yaml) :
queue:
topic: mytopic
exchange: myexchange
In rabbitmq, consumer are associated with exchanges. It allow you to define how the messages must be consumed (are all consumer listen to all message ? Is this enought if only one consumer read the message ? ...)

Here is an example of how to listen to a specific 'queue' using Spring Integration:
SpringIntegrationConfiguration.java
#Configuration
public class SpringIntegrationConfiguration {
#Value("${rabbitmq.queueName}")
private String queueName;
#Bean
public IntegrationFlow ampqInbound(ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Amqp.inboundAdapter(connectionFactory, queueName))
.handle(System.out::println)
.get();
}
}
ApplicationConfiguration.java
#Configuration
public class ApplicationConfiguration {
#Value("${rabbitmq.topicExchangeName}")
private String topicExchangeName;
#Value("${rabbitmq.queueName}")
private String queueName;
#Value("${rabbitmq.routingKey}")
private String routingKey;
#Bean
Queue queue() {
return new Queue(queueName, false);
}
#Bean
TopicExchange exchange() {
return new TopicExchange(topicExchangeName);
}
#Bean
Binding binding(Queue queue, TopicExchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with(routingKey);
}
}
Application.yml
rabbitmq:
topicExchangeName: spring-boot-exchange
queueName: spring-boot
routingKey: foo.bar.#

Spring integration error channel when using PollableChannel

We're using Spring integration in my application. I'd like put some objects into channel for asynchronous processing and error handling. So for this, I configured MessageGateway with error channel and PollableChannel for handling objects to be processed.
The problem
So I'm calling messageGateway.processMessage(message) to put message into channel. This works as expected - calling this method is non-blocking, messages get processed and are forwarded to next channel. But when processing method throws an exception, it is not redirected to error channel.
On the other hand when I change my processing channel from PollableChannel to SubscribableChannel, error channel works as expected, but calling the gateway is of course blocking. What am I missing? Can I have both non blocking call and error channel?
The code
Component doing the message processing:
#Component
public MessageProcessor {
#Transactional
#ServiceActivator(inputChannel = "msg.process", outputChannel = "msg.postprocess")
public void processMessage(MyMessage message) {
// Message processing that may throw exception
}
}
Channel definition:
#Configuration
public class IntegrationConfig {
#Bean(name = "msg.process")
private MessageChannel processChannel() {
return new RendezvousChannel();
}
#Bean(name = "msg.error")
private MessageChannel errorChannel() {
return new DirectChannel();
}
}
My gateway looks like this:
#MessagingGateway(errorChannel = "msg.error")
public interface MessagingGateway {
#Gateway(requestChannel = "msg.processing")
void processMessage(MyMessage message);
}
Error handler:
#Component
public ErrorHandlers {
#Transactional
#ServiceActivator(inputChannel = "msg.error")
public void processError(MessagingException me) {
// Error handling is here
}
}

But when processing method throws an exception, it is not redirected to error channel.
When a gateway method returns void, the calling thread is released immediately when it returns to the gateway.
The gateway does not add an error channel header in this case (in the next release - 5.0) we have changed that.
In the meantime, you can use a header enricher to set the errorChannel header to your error channel. You can also use the defaultHeaders property on the #MessagingGateway - see the comments on this answer.