Develop Reference

Resilience4j circuit breaker does not switch back to closed state from half open after the downstream system is up again

I have two micro-services, order-service, and inventory-service. The order-service makes a call to inventory-service to check if ordered items are in stock. An order is placed only if all the items in the order request are in stock. If the inventory-service is down or slow, the circuit breaker part triggers. The case of an order not being placed because of a missing item is not a failure case for the circuit breaker. This works well as intended only until the circuit has never switched from closed state to half open state.
What I mean by that is if more than 5 consecutive orders cannot be placed because of a missing item, the circuit does not switch to open state. This is as expected. If the inventory-service is brought down, 3 more failed requests cause the circuit to move to open and subsequently to half open state. This also is as expected. However when the inventory-service comes up again, and the only requests that are made are requests containing one or more items not in stock, the circuit remains in half_open state continuously. This is not ok. A missing item in an order is a success case and should increment the successful buffered call count, but it doesn't. Looking at the actuator info, it looks like these calls are not counted either as failure or as success cases.
What am I doing wrong.
Note -- if I make sufficient number of calls where order gets placed, then the circuit switches to closed again. That's ok but shouldn't the case of ignored exception count as a success case even if the only calls that are made are those with one or missing items.
Following are the properties of my circuit breaker in the calling microservice.
resilience4j.circuitbreaker.instances.inventory.register-health-indicator=true
resilience4j.circuitbreaker.instances.inventory.event-consumer-buffer-size=10
resilience4j.circuitbreaker.instances.inventory.sliding-window-type=COUNT_BASED
resilience4j.circuitbreaker.instances.inventory.sliding-window-size=5
resilience4j.circuitbreaker.instances.inventory.failure-rate-threshold=50
resilience4j.circuitbreaker.instances.inventory.wait-duration-in-open-state=5s
resilience4j.circuitbreaker.instances.inventory.permitted-number-of-calls-in-half-open-state=3
resilience4j.circuitbreaker.instances.inventory.automatic-transition-from-open-to-half-open-enabled=true
resilience4j.circuitbreaker.instances.inventory.ignore-exceptions=com.mayrevision.orderservice.exception.OrderItemNotFoundException
I have a custom exception handler for OrderItemNotFoundException.
#ControllerAdvice
#ResponseStatus
public class OrderItemNotFoundExceptionHandler extends ResponseEntityExceptionHandler {
#ExceptionHandler(OrderItemNotFoundException.class)
public ResponseEntity<ErrorResponse> getErrorMessage(OrderItemNotFoundException exception) {
ErrorResponse response = new ErrorResponse(HttpStatus.NOT_ACCEPTABLE, exception.getMessage());
return ResponseEntity.status(HttpStatus.NOT_ACCEPTABLE).body(response);
}
}
The controller code is --
#PostMapping
#ResponseStatus(HttpStatus.CREATED)
#CircuitBreaker(name = "inventory", fallbackMethod = "fallBackMethod")
public String placeOrder(#RequestBody OrderRequest orderRequest) throws OrderItemNotFoundException {
orderService.placeOrder(orderRequest);
return "Order placed successfully";
}
public String fallBackMethod(OrderRequest orderRequest, RuntimeException runtimeException) {
return "The order could not be placed. Please try back after some time.";
}
Edit -- Edited resilience4j.circuitbreaker.instances.inventory.ignore-exceptions[0]=com.mayrevision.orderservice.exception.OrderItemNotFoundException to resilience4j.circuitbreaker.instances.inventory.ignore-exceptions=com.mayrevision.orderservice.exception.OrderItemNotFoundException in application.properties.

Looks like this is how the exception mechanism in Resilience4j is designed to work. If I want the exception to be treated as a success case in all the cases (including the case mentioned above), I should catch it. So I changed the code as follows --
#PostMapping
#CircuitBreaker(name = "inventory", fallbackMethod = "fallBackMethod")
public CompletableFuture<ResponseEntity<StatusResponse>> placeOrder(#RequestBody OrderRequest orderRequest) {
return CompletableFuture.supplyAsync(() -> {
try {
String message = orderService.placeOrder(orderRequest);
StatusResponse response = new StatusResponse(HttpStatus.CREATED, message);
return ResponseEntity.status(HttpStatus.CREATED).body(response);
} catch (OrderItemNotFoundException e) {
return ResponseEntity.status(HttpStatus.NOT_ACCEPTABLE)
.body(new StatusResponse(HttpStatus.NOT_ACCEPTABLE, e.getMessage()));
}
});
}
public CompletableFuture<ResponseEntity<StatusResponse>> fallBackMethod(OrderRequest orderRequest, RuntimeException e) {
return CompletableFuture.supplyAsync(() -> {
//some logic
});
}
This is working close to expected.

nesting synchron and asynchron spring webclient requests without block()

i wrote a little piece of code as PoC to see if i can get that running. it should be a client for a website (oldschool chat system). unfortunately, they have some "special" handling of their website, e.g. frames and some calculations in .js that must be done before i can access the stream.
public void run() {
final URI chatStreamUri = getUserStreamUrl();
String block = webClient
.get()
.uri(chatStreamUri)
.retrieve()
.bodyToMono(String.class)
.map(s -> transformToBlah(s, chatAppUser.getInternalPass()))
.flatMap(s -> webClient.get().uri(s).retrieve().bodyToMono(String.class))
.map(this::getCrapStreamUrl)
.block();
System.out.println("block url: " + block);
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
webClient
.get()
.uri(block)
.retrieve()
.bodyToFlux(String.class)
.subscribe(writeTo,
new Consumer<Throwable>() {
#Override
public void accept(Throwable throwable) {
System.out.println(throwable);
}
}
);
}
unfortunately that throws the error in the sysout(throwable):
java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-epoll-2
I only have the .block() in the synchronous part so i assume it is because of that.
the steps that must happen are:
make a GET on the streamUrl (i guess that sets some cookies or so).
that response contains some wierdo looking url like http://example.com/user/+pass+/stream, where i replace that +pass+ with the password of that user
make a GET on the http://example.com/user/$verySecret/stream url to get the token - that is somehow generated on server side, so the response looks like: "the stream url is: http://example.com/user/24987624/blabla/stream"
and then listen on that http://example.com/user/24987624/blabla/stream for content
all those calls must be done on the same client object, i dont know why but when i access the stream url directly, it will fail (guess some cookie stuff).
anyone an idea how to achieve that with springs reactive webclient? or should i consider something else? maybe its not the correct library to use. thought i take a look at that fancy new reactive stuff ;)
btw.: when i debug, it works. thats why i added that ugly Thread.sleep() which unfortunately did not workaround the problem.
edit:
when i try it with this (as suggested i guess) it does not work:
webClient
.get()
.uri(chatStreamUri)
.retrieve()
.bodyToFlux(String.class)
.map(s -> transformToBlah(s, chatAppUser.getInternalPass()))
.flatMap(s -> webClient.get().uri(s).retrieve().bodyToMono(String.class))
.map(this::getCrapStreamUrl)
.subscribe(writeTo, new Consumer<Throwable>() {
#Override
public void accept(Throwable throwable) {
System.out.println(throwable);
}
});
then i only receive a message like:
java.lang.RuntimeException: got but cant work with that
this is from one of my methods that expect not a <html> but a text block containing way more text (like a window.replace code snippet -> thats where i filter out the next url i have to access).
when i debug that and set a breakpoint, i really only retrieve a <html> tag, nothing more
edit #2: when i use bodyToMono instead i get more stuff (makes sense now when i think about it), but still, the methods are not called as expected. Now the first method is called (transformToBlah()), then the second method is called which extracts the stream url, but then there is no subscribe on that extracted stream url, what happens is that the subscriber retrieves the stream url. thats not what i expected. for me, it looks like i have to now somehow call again webclient.get().uri($theResultFromMapThis::getCrapStreamUrl).subscribe(writeTo);
what does also not work is something like:
webClient
.get()
.uri(chatStreamUri)
.retrieve()
.bodyToMono(String.class)
.map(s -> transformToBlah(s, chatAppUser.getInternalPass()))
.flatMap(s -> webClient.get().uri(s).retrieve().bodyToMono(String.class))
.map(this::getCrapStreamUrl)
.flatMap(s -> webClient.get().uri(s).retrieve().bodyToMono(String.class))
.subscribe(writeTo, new Consumer<Throwable>() {
#Override
public void accept(Throwable throwable) {
System.out.println(throwable);
}
});
edit #3: what a heck of a ride. here is the code that works (a while):
webClient
.get()
.uri(chatStreamUri)
.retrieve()
.bodyToMono(String.class)
.flatMap(s -> Mono.just(transformToBlah(s, chatAppUser.getInternalPass())))
.flatMap(x -> webClient.get().uri(x).retrieve().bodyToMono(String.class))
.map(this::getCrapStreamUrl)
.flatMapMany(s -> webClient.get().uri(s).retrieve().bodyToFlux(String.class))
.subscribe(writeTo, new Consumer<Throwable>() {
#Override
public void accept(Throwable throwable) {
System.out.println("err: " + throwable);
}
});
edit #4: i first thought the above code made it, and it works for some time, but then i again retrieve:
java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-epoll-2 from the System.out.println and no idea why :/
edit #5: seems like those exceptions are from a very different call, that has nothing to do with that part at all. i will open another thread for that. commented that other part/code and this works now as expected for about an hour, so guess the code works. thanks!
i will watch that application now for a couple of hours and if it still does work i close this.

Handling exception in WebClient throws io.netty.handler.timeout.ReadTimeoutException

So I am new to Reactive programming and I wrote some code that I would like to test. Those are more of a integration tests as I am live copying files and later check if they are the same. I have a MockWebServer mocking my reponse to be 4xx which is handled well in the code. Unfortunately, I am also getting io.netty.handler.timeout.ReadTimeoutException which covers up my custom WebClientResponseException so in the test I am getting the wrong exception. Basically I have two questions, why on earth am I getting this io.netty.handler.timeout.ReadTimeoutException exception? It appears only after doOnError() method for some reason and I am not sure why is it even happening.
Right now code is at it is and it's being synchronous, I am well aware of that.
Second question is, how could I handle my custom exception in the tests after a given number of retries? Right now it is 3 and only then I would like my other exception to be thrown.
Here is the code:
AsynchronousFileChannel fileChannel = AsynchronousFileChannel.open(targetPath, StandardOpenOption.WRITE);
Flux<DataBuffer> fileDataStream = Mono.just(filePath)
.map(file -> targetPath.toFile().exists() ? targetPath.toFile().length() : 0)
.map(bytes -> webClient
.get()
.uri(uri)
.accept(MediaType.APPLICATION_OCTET_STREAM)
.header("Range", String.format("bytes=%d-", bytes))
.retrieve()
.onStatus(HttpStatus::is4xxClientError, clientResponse -> Mono.error(new CustomException("4xx error")))
.onStatus(HttpStatus::is5xxServerError, clientResponse -> Mono.error(new CustomException("5xx error")))
.bodyToFlux(DataBuffer.class)
.doOnError(throwable -> log.info("fileDataStream onError", throwable))
)
.flatMapMany(Function.identity());
return DataBufferUtils
.write(fileDataStream, fileChannel)
.map(DataBufferUtils::release)
.doOnError(throwable -> {
try {
fileChannel.force(true);
} catch (IOException e) {
throw new WritingException("failed force update to file channel", e);
}
})
.retry(3)
.doOnComplete(() -> {
try {
fileChannel.force(true);
} catch (IOException e) {
log.warn("failed force update to file channel", e);
throw new WritingException("failed force update to file channel", e);
}
})
.doOnError(throwable -> targetPath.toFile().delete())
.then(Mono.just(target));
The response is Mono<Path> as I am interested only in the Path of the newly created and copied file.
Any comments regarding code are welcome.
The copying mechanism was made basing on this thread Downlolad and save file from ClientRequest using ExchangeFunction in Project Reactor

So basically the problem was in tests. I had a MockResponse enqueued to the MockWebServer only once so upon retrying in the WebClient mocked server did not have any response set (basically it behaved like it is not available at all as there was no mocked response).
To be able to handle exceptions in case of the server being completely down it is in my opinion worth adding lines that goes something like this into your flux chain:
.doOnError(ChannelException.class, e -> {
throw new YourCustomExceptionForHandlingServerIsDownSituation("Server is unreachable", e);
})
This will help you handle ReadTimeoutException from Netty (in case of server being not reachable) as it extends ChannelException class. Always handle your exceptions.

How to handle exceptions in SpringBoot ListenableFuture

so I have a SpringBoot end point controller that starts like this:
#RequestMapping(value = "/post", method = RequestMethod.POST, produces = MediaType.APPLICATION_JSON_VALUE)
public Response post(#Valid #RequestBody Message message) throws FailedToPostException {
message.setRecieveTime(System.currentTimeMillis());
return this.service.post(message);
}
And the post function:
public Response post(Message message) throws FailedToPostException{
ListenableFuture<SendResult<String, Message>> future = kafkaTemplate.send("topicName", message);
future.addCallback(new ListenableFutureCallback<SendResult<String, Message>>() {
#Override
public void onSuccess(SendResult<String, Message> result) {
LOGGER.info("Post Finished. '{}' with offset: {}", message,
result.getRecordMetadata().offset());
}
#Override
public void onFailure(Throwable ex) {
LOGGER.error("Message Post Failed. '{}'", message, ex);
long nowMillis = System.currentTimeMillis();
int diffSeconds = (int) ((nowMillis - message.getRecieveTime()) / 1000);
if (diffSeconds >= 10) {
LOGGER.debug("timeout sending message to Kafka, aborting.");
return;
}
else {
post(message);
}
}
});
LOGGER.debug("D: " + Utils.getMetricValue("buffer-available-bytes", kafkaTemplate));
return new Response("Message Posted");
}
Now you can see, that we are trying to make sure, if a kafkaTemplate.send failed, we are going to recursively invoke post(message) again for up to 10 seconds, until the producer memory buffer clears and the message gets through.
The problems are:
We want to be able to return failure response to the endpoint's client (eg: "Failed to acknowledge the message").
Is there any better way to handle exceptions from a Future in a piece of code like that above?
Is there a way to avoid using a recursive function here? We did that, because we wanted to attempt delivery of the message to Kafka for like 10 seconds, before sending it as an email to look at.
Side note: I still didnt use buffer-available-bytes attribute from kafkaTemplate.metrics(), I intend to use it to minimize the chance of this problem, but still need to handle the above just in case of some race conditions

There are a few ways to do this, but I really like Spring Retry as a way to solve this kind of problem. It's a bit of pseudo code here, but if you need more specifics on how to do it, I could make things more explicit:
#Retryable(maxAttempts = 10, value = KafkaSendException.class)
public Response post(Message message) throws FailedToPostException{
ListenableFuture<SendResult<String, Message>> future = kafkaTemplate.send("topicName", message);
try {
future.get(1. TimeUnit.SECONDS);
} catch(SomeException ex) {
LOGGER.error("Message Post Failed. '{}'", ex.getCause().getMessage(), ex);
throw ex;
}
LOGGER.info("Post Finished. '{}' with offset: {}", message,
result.getRecordMetadata().offset());
}
Effectively does the same thing without recursing. I wouldn't recommend recursing code for error handling.
The controller should be able to massage the actual KafkaSendException with a nice #ExceptionHandler.

Is this a bad practice?

Is the following code considered a bad practice? Do you think it can be done otherwise?
The goal is to always update the status, either with success (i.e invocation to service.invoke(id);returns normally ) or with failure...
#Autowired
private Service service;
public void onMessage(Message message) {
String id = null;
String status = "FAILED";
try {
id = ((TextMessage) message).getText();
status = service.invoke(id); //can throw unchecked exception
} catch (final JMSException e) {
throw new RuntimeException(e);
} finally {
if (StringUtils.isNumeric(id)) {
service.update(id, status);
}
}
}

It depends on your Use-case, whether you have to perform a step or not based on previous step. Using finally may execute your second step regardless what exception you may receive.
I would recommend having the second step outside try...catch block so that you'll update only when you have got any exception you've Expected and continue to your second step, else, your method will throw and exit.

i think you should not use implementation of message listener , you should wire them independent of spring tech . just pojo based . use <jms:listener-container > with <jms:listener>