Develop Reference

Spring Integration manually publish message to channel

I'm in the process of learning how to use the Java Spring Framework and started experimenting with Spring Integration. I'm trying to use Spring Integration to connect my application to an MQTT broker both to publish and subscribe to messages but I'm having trouble finding a way to manually publish messages to an outbound channel. If possible I want to build it using notations in the java code exclusively rather than xml files defining beans and other related configuration.
In every example I've seen the solution to manually publishing a message seems to be to use a MessagingGateway Interface and then use the SpringApplicationBuilder to get the ConfigurableApplicationContext to get a reference to the gateway interface in the main method. The reference is then used to publish a message. Would it be possible to use AutoWired for the interface instead? In my attempts I just get a NullPointer.
My aim is to build a game where I subscribe to a topic to get game messages and then whenever the user is ready to make the next move, publish a new message to the topic.
Update:
This is one of the examples I've been looking at of how to setup an outbound channel: https://docs.spring.io/spring-integration/reference/html/mqtt.html
Update 2 after answer from Gary Russel:
This is some example code I wrote after looking at examples which gets me a NullPointer when using #AutoWired for the Gateway when running gateway.sendToMqtt in Controller.java. What I want to achieve here is to send an mqtt message manually when a GET request is handled by the controller.
Application.java
#SpringBootApplication
public class Application {
public static void main(String[] args){
SpringApplication.run(Application.class, args);
}
}
Controller.java
#RestController
#RequestMapping("/publishMessage")
public class Controller {
#Autowired
static Gateway gateway;
#RequestMapping(method = RequestMethod.GET)
public int request(){
gateway.sendToMqtt("Test Message!");
return 0;
}
}
MqttPublisher.java
#EnableIntegration
#Configuration
public class MqttPublisher {
#Bean
public MqttPahoClientFactory mqttClientFactory(){
DefaultMqttPahoClientFactory factory = new DefaultMqttPahoClientFactory();
factory.setServerURIs("tcp://localhost:1883");
return factory;
}
#Bean
#ServiceActivator(inputChannel = "mqttOutboundChannel")
public MessageHandler mqttOutbound(){
MqttPahoMessageHandler messageHandler =
new MqttPahoMessageHandler("clientPublisher", mqttClientFactory());
messageHandler.setAsync(true);
messageHandler.setDefaultTopic("topic");
return messageHandler;
}
#Bean
public MessageChannel mqttOutboundChannel(){
return new DirectChannel();
}
#MessagingGateway(defaultRequestChannel = "mqttOutboundChannel")
public interface Gateway {
void sendToMqtt(String data);
}
}
Update:
Not sure if this is the proper logging but it is what I get from adding:
logging.level.org.springframework.web=Debug
logging.level.org.hibernate=Error
to application.properties.
https://hastebin.com/cuvonufeco.hs

Use a Messaging Gateway or simply send a message to the channel.
EDIT
#SpringBootApplication
public class So47846492Application {
public static void main(String[] args) {
SpringApplication.run(So47846492Application.class, args).close();
}
#Bean
public ApplicationRunner runner(MyGate gate) {
return args -> {
gate.send("someTopic", "foo");
Thread.sleep(5_000);
};
}
#Bean
#ServiceActivator(inputChannel = "toMqtt")
public MqttPahoMessageHandler mqtt() {
MqttPahoMessageHandler handler = new MqttPahoMessageHandler("tcp://localhost:1883", "foo",
clientFactory());
handler.setDefaultTopic("myTopic");
handler.setQosExpressionString("1");
return handler;
}
#Bean
public MqttPahoClientFactory clientFactory() {
DefaultMqttPahoClientFactory factory = new DefaultMqttPahoClientFactory();
factory.setUserName("guest");
factory.setPassword("guest");
return factory;
}
#Bean
public MqttPahoMessageDrivenChannelAdapter mqttIn() {
MqttPahoMessageDrivenChannelAdapter adapter =
new MqttPahoMessageDrivenChannelAdapter("tcp://localhost:1883", "bar", "someTopic");
adapter.setOutputChannelName("fromMqtt");
return adapter;
}
#ServiceActivator(inputChannel = "fromMqtt")
public void in(String in) {
System.out.println(in);
}
#MessagingGateway(defaultRequestChannel = "toMqtt")
public interface MyGate {
void send(#Header(MqttHeaders.TOPIC) String topic, String out);
}
}

Spring Integration: Persistent and transactional QueueChannel

In Spring Integration we have a Setup that looks something like this:
--->
--->
(dispatcher) Messages --> Gateway ----> QueueChannel ---> MessageHandler (worker)
--->
--->
So we have one Dispatcher Thread that takes Messages from a MQTT-Broker and forwards them into the Queue. The Poller for the Queue is provided with a TaskExecuter, so the Consumer is multithreaded.
We managed to implement all the functionalities. So the just described setup is already implemented.
Now to guarantee no data loss we want to make two things:
1.:
We want our queue to persist the data, so when the Programm shuts down ungracefully, all the data in the queue will still be there.
This also worked for us, we are using MongoDB as a database because we read somewhere in your docs that this is the recommended way to do it.
2.:
The second thing we want to assure is that the worker threads are working transactional. So only if the worker threads return correctly the messages will permanently be deleted from the queue (and therefore the persistent MessageStore). If the program shuts down during the processing of a message (by the worker thread) the message will still be in the queue at the next startup.
Also if the worker, for example, thows an exception during the processing of the message, it will be put back into the queue.
Our implementation:
As explained before, the basic setup of the program is already implemented. We then extended the basic implementation with a message store implementation for the queue.
QueueChannel:
#Bean
public PollableChannel inputChannel(BasicMessageGroupStore mongoDbChannelMessageStore) {
return new QueueChannel(new MessageGroupQueue(mongoDbChannelMessageStore, "inputChannel"));
}
backed by a Messagestore:
#Bean
public BasicMessageGroupStore mongoDbChannelMessageStore(MongoDbFactory mongoDbFactory) {
MongoDbChannelMessageStore store = new MongoDbChannelMessageStore(mongoDbFactory);
store.setPriorityEnabled(true);
return store;
}
the matching Poller:
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
PollerMetadata poll = Pollers.fixedDelay(10).get();
poll.setTaskExecutor(consumer);
return poll;
}
Executor:
private Executor consumer = Executors.newFixedThreadPool(5);
What we have tried?
As explained now we want to extend this implementation with a transactional functionality. We tried using the setTransactionSynchronizationFactory like explained here but it wasn't working (didn't get errors or anything but the behavior was still as it was before we added the TransactionSynchronizer):
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
PollerMetadata poll = Pollers.fixedDelay(10).get();
poll.setTaskExecutor(consumer);
BeanFactory factory = mock(BeanFactory.class);
ExpressionEvaluatingTransactionSynchronizationProcessor etsp = new ExpressionEvaluatingTransactionSynchronizationProcessor();
etsp.setBeanFactory(factory);
etsp.setAfterRollbackChannel(inputChannel());
etsp.setAfterRollbackExpression(new SpelExpressionParser().parseExpression("#bix"));
etsp.setAfterCommitChannel(inputChannel());
etsp.setAfterCommitExpression(new SpelExpressionParser().parseExpression("#bix"));
DefaultTransactionSynchronizationFactory dtsf = new DefaultTransactionSynchronizationFactory(etsp);
poll.setTransactionSynchronizationFactory(dtsf);
return poll;
}
What would be the best way to realize our requirements in spring integration?
EDIT:
As recommended in the answer I chose to do this with the JdbcChannelMessageStore. So I tried converting the XML Implementation described here (18.4.2) into Java. I wasn't quite sure on how to do it, this is what I have tried so far:
I created H2 database and run the script shown here on it.
Created JDBCChannelMessageStore Bean:
#Bean
public JdbcChannelMessageStore store() {
JdbcChannelMessageStore ms = new JdbcChannelMessageStore();
ms.setChannelMessageStoreQueryProvider(queryProvider());
ms.setUsingIdCache(true);
ms.setDataSource(dataSource);
return ms;
}
Created H2ChannelMessageStoreQueryProvider
#Bean
public ChannelMessageStoreQueryProvider queryProvider() {
return new H2ChannelMessageStoreQueryProvider();
}
Adapted the poller:
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() throws Exception {
PollerMetadata poll = Pollers.fixedDelay(10).get();
poll.setTaskExecutor(consumer);
poll.setAdviceChain(Collections.singletonList(transactionInterceptor()));
return poll;
}
Autowired my PlaatformTransactionManager:
#Autowired
PlatformTransactionManager transactionManager;
And created TransactionInterceptor from the TransactonManager:
#Bean
public TransactionInterceptor transactionInterceptor() {
return new TransactionInterceptorBuilder(true)
.transactionManager(transactionManager)
.isolation(Isolation.READ_COMMITTED)
.propagation(Propagation.REQUIRED)
.build();
}

If you need to have queue as transactional, you definitely should take a look into the transactional MessageStore. And only JDBC one is like that. Just because only JDBC support transactions. So, when we perform DELETE, it is OK only if TX is committed.
The MongoDB, nor any other NoSQL DataBases, support such a model, therefore you only can push back the failed messages to the DB on rollback using TransactionSynchronizationFactory.
UPDATE
#RunWith(SpringRunner.class)
#DirtiesContext
public class So47264688Tests {
private static final String MESSAGE_GROUP = "transactionalQueueChannel";
private static EmbeddedDatabase dataSource;
#BeforeClass
public static void init() {
dataSource = new EmbeddedDatabaseBuilder()
.setType(EmbeddedDatabaseType.H2)
.addScript("classpath:/org/springframework/integration/jdbc/schema-drop-h2.sql")
.addScript("classpath:/org/springframework/integration/jdbc/schema-h2.sql")
.build();
}
#AfterClass
public static void destroy() {
dataSource.shutdown();
}
#Autowired
private PollableChannel transactionalQueueChannel;
#Autowired
private JdbcChannelMessageStore jdbcChannelMessageStore;
#Autowired
private PollingConsumer serviceActivatorEndpoint;
#Autowired
private CountDownLatch exceptionLatch;
#Test
public void testTransactionalQueueChannel() throws InterruptedException {
GenericMessage<String> message = new GenericMessage<>("foo");
this.transactionalQueueChannel.send(message);
assertTrue(this.exceptionLatch.await(10, TimeUnit.SECONDS));
this.serviceActivatorEndpoint.stop();
assertEquals(1, this.jdbcChannelMessageStore.messageGroupSize(MESSAGE_GROUP));
Message<?> messageFromStore = this.jdbcChannelMessageStore.pollMessageFromGroup(MESSAGE_GROUP);
assertNotNull(messageFromStore);
assertEquals(message, messageFromStore);
}
#Configuration
#EnableIntegration
public static class ContextConfiguration {
#Bean
public PlatformTransactionManager transactionManager() {
return new DataSourceTransactionManager(dataSource);
}
#Bean
public ChannelMessageStoreQueryProvider queryProvider() {
return new H2ChannelMessageStoreQueryProvider();
}
#Bean
public JdbcChannelMessageStore jdbcChannelMessageStore() {
JdbcChannelMessageStore jdbcChannelMessageStore = new JdbcChannelMessageStore(dataSource);
jdbcChannelMessageStore.setChannelMessageStoreQueryProvider(queryProvider());
return jdbcChannelMessageStore;
}
#Bean
public PollableChannel transactionalQueueChannel() {
return new QueueChannel(new MessageGroupQueue(jdbcChannelMessageStore(), MESSAGE_GROUP));
}
#Bean
public TransactionInterceptor transactionInterceptor() {
return new TransactionInterceptorBuilder()
.transactionManager(transactionManager())
.isolation(Isolation.READ_COMMITTED)
.propagation(Propagation.REQUIRED)
.build();
}
#Bean
public TaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(5);
return threadPoolTaskExecutor;
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return Pollers.fixedDelay(10)
.advice(transactionInterceptor())
.taskExecutor(threadPoolTaskExecutor())
.get();
}
#Bean
public CountDownLatch exceptionLatch() {
return new CountDownLatch(2);
}
#ServiceActivator(inputChannel = "transactionalQueueChannel")
public void handle(Message<?> message) {
System.out.println(message);
try {
throw new RuntimeException("Intentional for rollback");
}
finally {
exceptionLatch().countDown();
}
}
}
}

Thanks to Artem Bilan for your great support. I finally found the solution. It seemed like there was another bean with the name transactionManager and transactionInterceptor active. This resulted in the strange behavior, that my trans-manager was never initialized, instead the other transactionmanager (null) was used for the transactioninterceptor and the PollingConsumer. Thats why my Transactionmanager in PollingConsumer was null, and why my Transactions were never working.
The solution was to rename all my beans, for some beans I also used the annotation #Primary to tell spring to always use this speciffic bean when autowired.
I also downgraded two 4.3, just to make sure this wasn't an error related to Version 5. I haven't testet if it would work with V 5 yet, but I think it should work also.

Multiple dynamic HTTP endpoints

I want to run multiple HTTP endpoints which should be creates based on list of paths.
Currently I'm able to create one endpoint:
#MessagingGateway(defaultRequestChannel = "requestChannel")
public interface Gateway {
String sendReceive(String in);
}
#Bean
public MessageChannel requestChannel() {
return new DirectChannel();
}
#Bean
public IntegrationFlow flow() {
return IntegrationFlows.from("requestChannel").transform(new ObjectToStringTransformer())
.handle(new MyHandle())
.get();
}
#Bean
public HttpRequestHandlingMessagingGateway httpGate() {
HttpRequestHandlingMessagingGateway gateway = new HttpRequestHandlingMessagingGateway(true);
RequestMapping mapping = new RequestMapping();
mapping.setMethods(HttpMethod.POST);
mapping.setPathPatterns("/path");
gateway.setRequestMapping(mapping);
gateway.setRequestChannel(requestChannel());
gateway.setRequestPayloadType(byte[].class);
return gateway;
}
but I want to do somthing like this:
#Autowired
List<String> paths;
#PostConstruct
public void createEndpoints() {
for (String path : paths) {
//code for dynamic endpoint creation
}
}
private class MyHandle extends AbstractReplyProducingMessageHandler {
#Override
protected Object handleRequestMessage(Message<?> requestMessage) {
return this.getMessageBuilderFactory().withPayload("Your message: " + requestMessage.getPayload());
}
}
Can you tell me how can I do it?

Since Java DSL 1.2 there is a IntegrationFlowContextexactly for such a use-case to register IntegrationFlow and dependent beans dynamically.
https://spring.io/blog/2016/09/27/java-dsl-for-spring-integration-1-2-release-candidate-1-is-available
The GA release today.
You should just follow with the samples in those blog post and pay attention to the org.springframework.integration.dsl.http.Http factory.
But, indeed, do that as early as possible. The #PostConstruct is good phase for this use-case.
When it will be later, the HandlerMapping won't be able to detect an new mapping. Just because it does the scan in its afterPropertiesSet().

Is it possible to set prefetch count on #RabbitListener

I know it is possible to make SimpleMessageListenerContainer bean and set prefetch count and message listener here, like this:
#Bean
public SimpleMessageListenerContainer messageListenerContainer(
ConnectionFactory rabbitConnectionFactory,
Receiver receiver) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(rabbitConnectionFactory);
container.setQueueNames("hello");
container.setMessageListener(new MessageListenerAdapter(receiver, "receive"));
container.setPrefetchCount(1000);
return container;
}
But how to set prefetch count for channel if I want to use declarative approach using #RabbitListener?
#Component
public class Receiver {
private static final Logger log = LoggerFactory.getLogger(Receiver.class);
#RabbitListener(queues = "hello") // how to set prefetch count here?
public void receive(String message) {
log.info(" [x] Received '{}'.", message);
}
}
It is not possible?

Solution according to #artem-bilan answer:
Declare RabbitListenerContainerFactory bean with prefetch count 10 in some #Configuration class:
#Bean
public RabbitListenerContainerFactory<SimpleMessageListenerContainer> prefetchTenRabbitListenerContainerFactory(ConnectionFactory rabbitConnectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(rabbitConnectionFactory);
factory.setPrefetchCount(10);
return factory;
}
Receiver bean uses this factory bean:
#Component
public class Receiver {
private static final Logger log = LoggerFactory.getLogger(Receiver.class);
#RabbitListener(queues = "hello", containerFactory = "prefetchTenRabbitListenerContainerFactory")
public void receive(String message) {
log.info(" [x] Received '{}'.", message);
}
#RabbitListener(queues = "hello")
public void receiveWithoutPrefetch(String message) {
log.info(" [x] Received without prefetch '{}'.", message);
}
}
Two listeners here is just for demo purpose.
With this configuration Spring creates two AMQP channels. One for each #RabbitListener. First with prefetch count 10 using our new prefetchTenRabbitListenerContainerFactory bean and second with prefetch count 1 using default rabbitListenerContainerFactory bean.

The #RabbitListener has containerFactory option:
/**
* The bean name of the {#link org.springframework.amqp.rabbit.listener.RabbitListenerContainerFactory}
* to use to create the message listener container responsible to serve this endpoint.
* <p>If not specified, the default container factory is used, if any.
* #return the {#link org.springframework.amqp.rabbit.listener.RabbitListenerContainerFactory}
* bean name.
*/
String containerFactory() default "";
Where you can configure SimpleRabbitListenerContainerFactory with the desired prefetchCount and the target SimpleMessageListenerContainer for that annotation will have that option for you.

I'm using Spring boot 2.3.3 and I changed the following in application.properties and it worked.
spring.rabbitmq.listener.direct.prefetch=1000
spring.rabbitmq.listener.simple.prefetch=1000
I don't know the difference between direct and simple so I set both.

If you want to use different prefetch for each queue you can create listener factory with default spring setting and override just prefetch.
#Bean
public RabbitListenerContainerFactory<SimpleMessageListenerContainer> prefetchRabbitListenerContainerFactory(SimpleRabbitListenerContainerFactoryConfigurer configurer, ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setPrefetchCount(prefetch);
return factory;
}

#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
int concurrentConsumers = Integer.parseInt(PropertiesLoader.getProperty("queue.concurrent-consumers"));
int maxConcurrentConsumers = Integer.parseInt(PropertiesLoader.getProperty("queue.max-concurrent-consumers"));
int consecutiveActiveTrigger = Integer.parseInt(PropertiesLoader.getProperty("queue.consecutive-active-trigger"));
int prefectCount = Integer.parseInt(PropertiesLoader.getProperty("queue.prefetch_count"));
factory.setPrefetchCount(prefectCount);
factory.setConcurrentConsumers(concurrentConsumers);
factory.setMaxConcurrentConsumers(maxConcurrentConsumers);
factory.setConsecutiveActiveTrigger(consecutiveActiveTrigger);
return factory;
}

How to create multiple channels with different receivers? Spring Redis pub/sub

I'm using Redis Publish Subscribe through Spring Data, but I'm having problems to add more than 1 Channel.
Currently I'm following the typical examples where the MessageListenerContainer is configured by adding a MessageListenerAdapter which has associated a Receiver class, as following:
The previous works perfectly and I'm able to push and receive messages.
However one I try to add a second listener adapter for creating a "channel with a differente receiver and I'm getting a NullPointerException.
The error is attached below. Is there a different way to adding a new adapter? In general I would like to add channels dynamically.
It is possible to add multiple channels associated with one specific receiver by providing a PatternTopic list at addMessageListener method.
Thanks for your help

I believe there is an important bug with Spring Redis when adding MessageListenerAdapter.
If the Receiver class does not extend from MessageListener (and so, implements onMessage) the inner method MethodInvoker() from MessageListenerAdapter class specifically ask if the Receiver is instance of MessageListener (see last line of image below).
To solve this problem just extend from MessageListener and then you can add the additional adapters directly.
It's a shame that spring-data-redis team do not enable issues in their github page to publish this bug. https://github.com/spring-projects/spring-data-redis

If anyone is still looking, use below configuration for Spring Boot 1.5.X
RedisConfig class for multiple channels:
#Configuration
public class RedisConfig {
#Bean
RedisMessageListenerContainer container(RedisConnectionFactory connectionFactory,
#Qualifier("notificationListenerAdapter") MessageListenerAdapter notificationListenerAdapter,
#Qualifier("countListenerAdapter") MessageListenerAdapter countListenerAdapter) {
RedisMessageListenerContainer container = new RedisMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.addMessageListener(notificationListenerAdapter, new PatternTopic("notification"));
container.addMessageListener(countListenerAdapter, new PatternTopic("count"));
return container;
}
#Bean("notificationListenerAdapter")
MessageListenerAdapter notificationListenerAdapter(RedisReceiver redisReceiver) {
return new MessageListenerAdapter(redisReceiver, "receiveNotificationMessage");
}
#Bean("countListenerAdapter")
MessageListenerAdapter countListenerAdapter(RedisReceiver redisReceiver) {
return new MessageListenerAdapter(redisReceiver, "receiveCountMessage");
}
#Bean
RedisReceiver receiver() {
return new RedisReceiver();
}
#Bean
StringRedisTemplate template(RedisConnectionFactory connectionFactory) {
return new StringRedisTemplate(connectionFactory);
}
}
RedisReceiver to receive messages from channels.
Note: Make sure the method name matches the method names defined above.
public class RedisReceiver {
private static final Logger LOGGER = LoggerFactory.getLogger(RedisReceiver.class);
public void receiveNotificationMessage(String message) {
LOGGER.info("Message Received from notification channel: <" + message + ">");
}
public void receiveCountMessage(String message) {
LOGGER.info("Message Received from count channel: <" + message + ">");
}
}
Test the flow:
public class TestMessages {
private static final Logger LOG = LoggerFactory.getLogger(TestMessages.class);
private final StringRedisTemplate redisTemplate;
public TestMessages(StringRedisTemplate redisTemplate) {
this.redisTemplate = redisTemplate;
}
public void sendNotification(String message) {
redisTemplate.convertAndSend("notification", message);
}
public void sendCount(String message) {
redisTemplate.convertAndSend("count", message);
}
}