Develop Reference

Spring-Kafka 2.6.5 infinite retry policy via stateful retry and SeekToCurrentErrorHandler

As the title suggests, I'm using spring-kafka version 2.6.5. In my architecture, I have a main topic that has a SimpleRetryPolicy paired with an ExponentialBackoffPolicy. If retry attempts are exhausted, I have a RecoveryCallback which sends the message to an error topic. The error topic is where my issues reside.
In this error topic, I need to be able to perform infinite retries and not let any messages be dropped. By 'dropped', I mean that if Spring crashes or something else equally bad occurs, I need to make sure that, when brought back up, any messages that are in the middle of processing can be re-polled (order doesnt matter). Basically I think I need to configure the ACKs so that they're confirmed after processing is done. As for infinitely retrying, I've searched around and found a number of helpful pieces of advice from users like Gary Russell. Unfortunately, differing spring-kafka versions and deprecations have made it a bit difficult to piece together a clear solution for my needs and version.
Currently, my setup looks as such:
#KafkaListener(topics = "my_topic",
groupId = "my_group_id",
containerFactory = "kafkaErrorListenerContainerFactory")
public void listenErrorTopic(String message) throws Exception {
processingLogic(message);
// Do I need to manually ACK afterwards (and thus also include additional params to access needed
// message components)?
}
#Bean
public ConsumerFactory<String, String> consumerFactory() {
Map<String, Object> props = new HashMap();
...
// Basing the need for the below 2 props off of previously found posts
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
// Unsure if the below prop is needed
// props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed")
...
return new DefaultKafkaConsumerFactory<>(props);
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaErrorListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new
ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
// A previous post said that infinite retries could only be achieved via state retry and STCEH,
// but there is an alternative in 2.6?
factory.setStatefulRetry(true);
// A previous post had '-1' passed to SeekToCurrentErrorHandler, but that is no longer possible.
// It was suggested instead to pass Long.MAX_VALUE to the backoff period for later versions, but the
// policy shown was a FixedBackOffPolicy.
factory.setErrorHandler(new SeekToCurrentErrorHandler());
RetryTemplate retryTemplate = new retryTemplate();
retryTemplate.setRetryPolicy(new AlwaysRetryPolicy());
// Do I need a recovery callback set in my RetryTemplate if I want it to be infinite?
ExponentialBackOffPolicy backoffPolicy = new ExponentialBackOffPolicy();
backOffPolicy.setInitialInterval(<props file value insertion here>)
backOffPolicy.setMultiplier(<props file value insertion here>)
backOffPolicy.setMaxInterval(<props file value insertion here>)
retryTemplate.setBackOffPolicy(backoffPolicy);
factory.setRetryTemplate(retryTemplate);
return factory;
}
Ideally I'd prefer Exponential over Fixed, but I'm mainly concerned with the ability to have it be done infinitely without max.interval.ms triggering a rebalance. I left comments in the code block where I have uncertainties. If someone could clarify things, it would be greatly appreciated!

Using stateful retry was specifically designed to be used with a STCEH to avoid a rebalance, before the STCEH supported back offs.
However, now that back off is supported in the STCEH, it is better to use that over a retry template.
If you use both, the actual retries is the multiple of the STCEH and retry template retries.
Now that the SeekToCurrentErrorHandler can be configured with a BackOff and has the ability to retry only certain exceptions (since version 2.3), the use of stateful retry, via the listener adapter retry configuration, is no longer necessary. You can provide the same functionality with appropriate configuration of the error handler and remove all retry configuration from the listener adapter. See Seek To Current Container Error Handlers for more information.
The configuration is much simpler.
You don't need to use manual acks; the container will commit the offsets based on the AckMode BATCH (default) or RECORD. The latter is more costly but provides less chance of redelivery.
For infinite retries, use a FixedBackOff with UNLIMITED_ATTEMPTS (Long.MAX_VALUE) in the maxAttempts property.
The ExponentialBackOff will retry infinitely by default. You just need to be sure that the maxInterval is less than the max.poll.interval.ms to avoid a rebalance.

Live reload on Database update

Previously I worked on Meteor and MongoDB. when I was working on it I noticed that Meteor server reloads the data if any changes happen in Mongo DB.
Can we do these things in Spring Boot, Java? I checked Live reload tools and plugins. These plugins and tools reload or restart the server when code changes, but not when DB is changed.

I assume that you are talking about MongoDB change streams.
Yes you can register a listener:
Imperative Style
Change stream events can be consumed using a MessageListener registered within a MessageListenerContainer. The container takes care of running the task in a separate Thread pushing events to the MessageListener.
#Configuration
class Config {
#Bean
MessageListenerContainer messageListenerContainer(MongoTemplate template) {
return new DefaultMessageListenerContainer(template);
}
}
Once the MessageListenerContainer is in place MessageListeners can be registered.
MessageListener<ChangeStreamDocument<Document>, Person> messageListener = (message) -> {
System.out.println("Hello " + message.getBody().getFirstname());
};
ChangeStreamRequest<Person> request = ChangeStreamRequest.builder()
.collection("person")
.filter(newAggregation(match(where("operationType").is("insert"))))
.publishTo(messageListener)
.build();
Subscription subscription = messageListenerContainer.register(request, Person.class);
// ...
Reactive Style
Change stream events be directly consumed via a Flux connected to the change stream.
Flux changeStream = reactiveTemplate
.changeStream(newAggregation(match(where("operationType").is("insert"))),
Person.class, ChangeStreamOptions.empty(), "person");
changeStream.doOnNext(event -> System.out.println("Hello " + event.getBody().getFirstname()))
.subscribe();
Read more about that:
https://github.com/spring-projects/spring-data-examples/tree/master/mongodb/change-streams

Spring transaction handling JMSTemplate inside #Transactional method

In our spring boot application we have a paging loop over the database which will send a JMS message for every page using a JMSTemplate. The method containing the loop is #Transactional. The JMSTemplate is created with the transacted flag set to true.
I have been browsing through the source code of JMSTemplate and as far as I can see it will not commit a transacted session if there is already a outer transaction going on but will put it into that transaction.
Let's now consider the following code:
#Transactional
public void loopThroughPages(String destination, String from, String to) {
Pageable pageRequest = PageRequest.of(0, maxPageSize);
Page<Event> eventPage;
do {
eventPage = eventRepo.getEventsInTimeRangeForDestination(from, to, destination, pageRequest);
if(eventPage.hasContent()) {
Message<String> eventMessage = buildEventMessage(eventPage.getContent());
JmsTemplate template = queueService.createTemplate(destination);
template.send(eventMessage);
pageRequest = eventPage.nextPageable();
}
} while(pageRequest != null && eventPage.hasNext());
}
The createTemplate creates a DynamicJmsTemplate using a CachingConnectionFactory and setSessionTransacted to true
I am now not entirely sure how this translates to the transactions. My understanding is that all N page messages are sent in the transaction created from the loopThroughPages and once the loopThroughPages method finishes it will commit all N messages and not after each message has been sent. That also implies that the transaction on MQ side will stay open until the last page has been processed. Is this understanding correct?

The key point here is the Transastion managment.
If you use a XA datasource and configure it on your spring boot app you will have a distributed transaction and the commit/rollback for your transaction will be managed by spring since that you have a method annotated with #Transactional otherwise you will have a local transaction managment and the transaction of your database and messaging system will be not syncronized.
For sending message you can by properties configure if the message are persisted or not that it means if your messages will be persisted by your messaging system, while for listener perspective you can configure the acknowledged mode.
By the way the my advice is let to spring managed the transaction and all will be fine but the real point of attention is if you want a distributed transaction between database and jms system you have only configure it atomikos can be an available option, otherwise it is sufficient do not manage the transaction by hand and let to Spring manage it for you.
I hope that it can help you

How to create #RabbitListener to be idempotent

Our configuration is: 1...n Message receivers with a shared database.
Messages should only be processed once.
#RabbitListener(bindings = #QueueBinding(
value = #Queue(value = "message-queue", durable = "true"),
exchange = #Exchange(value = TOPIC_EXCHANGE, type = "topic", durable = "true"),
key = MESSAGE_QUEUE1_RK)
)
public void receiveMessage(CustomMessage message) throws InterruptedException {
System.out.println("I have been received = " + message);
}
We want to to guarantee messages will be processed once, we have a message store with id's of messages already processed.
Is it possible to hook in this check before receiveMessage?
We tried to look at a MessagePostProcessor with a rabbitTemplate but didn't seem to work.
any advice on how to do this?
We tried with a MethodInterceptor and this works, but is pretty ugly.
Thanks
Solution found - thanks to Gary
I created a MessagePostProcessorInjector which implements SmartLifecycle
and on startup, I inspect each container and if it is a AbstractMessageListenerContainer add a customer MessagePostProccesser
and a custom ErrorHandler which looks for certain type of Exceptions and drops them (other forward to defaultErrorHandler)
Since we are using DLQ I found throwing exceptions or setting to null wouldn't really work.
I'll make a pull request to ignore null Messages after a MPP.

Interesting; the SimpleMessageListenerContainer does have a property afterReceivePostProcessors (not currently available via the listener container factory used by the annotation, but it could be injected later).
However, those postprocessors won't help because we still invoke the listener.
Please feel free to open a JIRA Improvement Issue for two things:
expose the afterReceivePostProcessors in the listener container factories
if a post processor returns null, skip calling the listener method.
(correction, the property is indeed exposed by the factory).
EDIT
How it works...
During context initialization...
For each annotation detected by the bean post processor the container is created and registered in the RabbitListenerEndpointRegistry
Near the end of context initialization, the registry is start()ed and it starts all containers that are configured for autoStartup (default).
To do further configuration of the container before it's started (e.g. for properties not currently exposed by the container factories), set autoStartup to false.
You can then get the container(s) from the registry (either as a collection or by id). Simply #Autowire the registry in your app.
Cast the container to a SimpleMessageListenerContainer (or alternatively a DirectMessageListenerContainer if using Spring AMQP 2.0 or later and you are using its factory instead).
Set the additional properties (such as the afterReceiveMessagePostProcessors); then start() the container.
Note: until we enhance the container to allow MPPs that return null, a possible alternative is to throw an AmqpRejectAndDontRequeueException from the MPP. However, this is probably not what you want if you have DLQs configured.

Throwing an exception extending from ImmediateAcknowledgeAmqpException from postProcessMessage() of DuplicateChecking MPP when message is duplicate will also not pass the message to the rabbit Listener.

Understanding Spring's Web Reactive Framework

I am currently developing an application with SpringBoot 2, spring-boot-starter-webflux on netty and jOOQ.
Below is the code that I have come up with after hours of research and stackoverflow searches. I have built in a lot of
logging in order to see what's happening on which thread.
UserController:
#RequestMapping(value = "/user", method = RequestMethod.POST)
public Mono<ResponseEntity<Integer>> createUser(#RequestBody ImUser user) {
return Mono.just(user)
.map(it -> {
logger.debug("Receiving request on thread: " + Thread.currentThread().getName());
return it;
})
.map(userService::create)
.map(it -> {
logger.debug("Sending response on thread: " + Thread.currentThread().getName());
return ResponseEntity.status(HttpStatus.CREATED).body(it);
})
.mapError(DuplicateKeyException.class, e -> new SomeSpecialException(e.getMessage(), e));
}
UserService:
public int create(ImUser user) {
return Mono.just(user)
.subscribeOn(Schedulers.elastic())
.map(u -> {
logger.debug("UserService thread: " + Thread.currentThread().getName());
return imUserDao.insertUser(u);
})
.block();
}
UserDao:
#Transactional(propagation = Propagation.REQUIRED, isolation = Isolation.READ_COMMITTED, rollbackFor = Exception.class)
public int insertUser(ImUser user) {
logger.debug("Insert DB on thread: " + Thread.currentThread().getName());
return dsl.insertInto(IM_USER,IM_USER.VERSION, IM_USER.FIRST_NAME, IM_USER.LAST_NAME, IM_USER.BIRTHDATE, IM_USER.GENDER)
.values(1, user.getFirstName(), user.getLastName(), user.getBirthdate(), user.getGender())
.returning(IM_USER.ID)
.fetchOne()
.getId();
}
The code works as expected, "Receiving request" and "Sending response" both run on the same thread (reactor-http-server-epoll-x)
while the blocking code ( the call to imUserDao.insertUser(u) ) runs on an elastic Scheduler thread (elastic-x).
The transaction is bound to the thread on which the annotated method is called (which is elastic-x) and thus works as expected (I have tested
it with a different method which is not posted here, to keep things simple).
Here is a log sample:
20:57:21,384 DEBUG admin.UserController| Receiving request on thread: reactor-http-server-epoll-7
20:57:21,387 DEBUG admin.UserService| UserService thread: elastic-2
20:57:21,391 DEBUG admin.ExtendedUserDao| Insert DB on thread: elastic-2
20:57:21,393 DEBUG tools.LoggerListener| Executing query
...
20:57:21,401 DEBUG tools.StopWatch| Finishing : Total: 9.355ms, +3.355ms
20:57:21,409 DEBUG admin.UserController| Sending response on thread: reactor-http-server-epoll-7
I have researched reactive programming for a long time now, but never quite got to program anything reactive. Now that I am, I am wondering if I am doing it correctly.
So here are my questions:
1. Is the code above a good way to handle incoming HTTP requests, query the DB and then respond?
Please ignore the logger.debug(...) calls which I have built in for the sake of my sanity :) I kind of expected to have a Flux< ImUser> as the argument to the controller method, in the sense that I have a stream of multiple potential requests
that will come at some point and will all be handled in the same way. Instead, the examples that I have found create a Mono.from(...); every time a request comes in.
2. The second Mono created in the UserService ( Mono.just(user) ) feels somewhat awkward. I understand that I need to start a new stream to be able to
run code on the elastic Scheduler, but isn't there an operator that does this?
3. From the way the code is written, I understand that the Mono inside the UserService will be blocked until the DB operation finishes,
but the original stream, which serves the requests, isn't blocked. Is this correct?
4. I plan to replace Schedulers.elastic() with a parallel Scheduler where I can configure the number of worker threads. The idea is that the number of maximum worker threads should be the same as maximum DB connections.
What will happen when all worker threads inside the Scheduler will be busy? Is that when backpressure jumps in?
5. I initially expected to have this code inside my controller:
return userService.create(user)
.map(it -> ResponseEntity.status(HttpStatus.CREATED).body(it))
.mapError(DuplicateKeyException.class, e -> new SomeSpecialException(e.getMessage(), e));
but I have not been able to achieve that AND keep the things running in the correct threads. Is there any way to achieve this inside my code?
Any help would be greatly appreciated. Thanks!

Service and Controller
The fact that your service is blocking is problematic, because then in the controller you are calling a blocking method from inside a map that isn't moved on a separate thread. This has the potential to block all controllers.
What you could do instead is return a Mono from UserService#create (remove the block() at the end). Since the service ensures that the Dao method call is isolated, it is less problematic. From there, no need to do Mono.just(user) in the Controller: just call create and start chaining operators directly on the resulting Mono:
#RequestMapping(value = "/user", method = RequestMethod.POST)
public Mono<ResponseEntity<Integer>> createUser(#RequestBody ImUser user) {
//this log as you saw was executed in the same thread as the controller method
logger.debug("Receiving request on thread: " + Thread.currentThread().getName());
return userService.create(user)
.map(it -> {
logger.debug("Sending response on thread: " + Thread.currentThread().getName());
return ResponseEntity.status(HttpStatus.CREATED).body(it);
})
.mapError(DuplicateKeyException.class, e -> new SomeSpecialException(e.getMessage(), e));
}
Logging
Note that if you want to log something there are a couple better options than doing a map and returning it:
doOnNext method is tailored for that: react to one of the reactive signals (in this instance, onNext: a value is emitted) and perform some non-mutating action, leaving the output sequence exactly the same as the source sequence. The "side-effect" of the doOn can be writing to the console or incrementing statistic counters for instance... There's also doOnComplete, doOnError, doOnSubscribe, doOnCancel, etc...
log simply logs all events in the sequence above it. It will detect if you use SLF4J and use the configured logger at DEBUG level if so. Otherwise it'll use the JDK Logging features (so you also need to configure that to display DEBUG level logs).
A word about transactions or rather anything relying on ThreadLocal
ThreadLocal and thread-stickiness can be problematic in reactive programming, because there's less guarantee of the underlying execution model staying the same throughout a whole sequence. A Flux can execute in several steps, each in a different Scheduler (and so thread or thread pool). Even at a specific step, one value could be processed by thread A of the underlying thread pool while the next one, arriving later on, would be processed on thread B.
Relying on Thread Local is less straightforward in this context, and we are currently actively working on providing alternatives that fit better in the reactive world.
Your idea of making a pool of the size of the connection pool is good, but not necessarily sufficient, with the potential of several threads being used by a transactional flux, thus maybe polluting some threads with the transaction.
What happens when a pool runs out of threads
If you are using a particular Scheduler to isolate blocking behavior like here, once it runs out of threads it would throw a RejectedExecutionException.