Develop Reference

How to invoke a retry of resilience4j on a spring mono in a spring-reactor and resilience4j when not using annotations

I'm calling a service class method using the transformDeferred method:
public Mono<SPathResponse> getPath(SPathRequest request) {
return pathService.getPath(request)
.transformDeferred(RetryOperator.of(retry));
}
My retry configuration:
Retry retry = Retry.of("es", RetryConfig.custom()
.maxAttempts(3)
.waitDuration(Duration.of(30, ChronoUnit.MILLIS))
.writableStackTraceEnabled(true)
.failAfterMaxAttempts(true)
.retryOnException(throwable -> throwable instanceof RuntimeException)
.build());
and the test method:
#Test
void shouldRetry() {
BDDMockito.given(pathService.getPath(any(SPathRequest.class)))
.willReturn(Mono.error(RuntimeException::new))
.willReturn(Mono.just(SPathResponse.builder().build()));
cachedPathService.getPath(SPathRequest.builder()
.sourceNodeId("2")
.sourceCategoryId("1234")
.destinationNodeId("123")
.build())
.as(StepVerifier::create)
.expectError(RuntimeException.class)
.verify();
var captor = ArgumentCaptor.forClass(SPathRequest.class);
BDDMockito.then(pathService).should(times(2)).getPath(captor.capture());
Running it, I do get the expected exception, but 'getPath' is invoked only once.
I probably miss something cause the retry mechanism should have retried and return the stubbed result on the 2nd invocation which should fail the test since no exception occurred and the actual result should have been expected.
What is wrong with my configuration?
Edit:
I want the equivalent of this snippet (from resilience4j-reactor examples) for directly invoking on my Mono rather then wrapping a function with Mono.fromCallable):
#Test
public void returnOnErrorUsingMono() {
RetryConfig config = retryConfig();
Retry retry = Retry.of("testName", config);
RetryOperator<String> retryOperator = RetryOperator.of(retry);
given(helloWorldService.returnHelloWorld())
.willThrow(new HelloWorldException());
StepVerifier.create(Mono.fromCallable(helloWorldService::returnHelloWorld)
.transformDeferred(retryOperator))
.expectSubscription()
.expectError(HelloWorldException.class)
.verify(Duration.ofSeconds(1));
StepVerifier.create(Mono.fromCallable(helloWorldService::returnHelloWorld)
.transformDeferred(retryOperator))
.expectSubscription()
.expectError(HelloWorldException.class)
.verify(Duration.ofSeconds(1));
then(helloWorldService).should(times(6)).returnHelloWorld();
Retry.Metrics metrics = retry.getMetrics();
assertThat(metrics.getNumberOfFailedCallsWithRetryAttempt()).isEqualTo(2);
assertThat(metrics.getNumberOfFailedCallsWithoutRetryAttempt()).isEqualTo(0);
}
Where retryConfig is defined like this:
private RetryConfig retryConfig() {
return RetryConfig.custom()
.waitDuration(Duration.ofMillis(10))
.build();
}
Thanks.

Handle exception after reaching max attempts in resilience4j-retry using Spring Boot

I have a scenario I want to log each retry attempt and when the last one fails (i.e. maxAttempts reached) a exception is thrown and let's say an entry to a database is created.
I try to achieve this using Resilience4j-retry with Spring Boot, therefore I use application.yml and annotations.
#Retry(name = "default", fallbackMethod="fallback")
#CircuitBreaker(name = "default", fallbackMethod="fallback")
public ResponseEntity<List<Person>> person() {
return restTemplate.exchange(...); // let's say this always throws 500
}
The fallback logs the cause of the exception into an application log.
public ResponseEntity<?> fallback(Exception e) {
var status = HttpStatus.INTERNAL_SERVER_ERROR;
var cause = "Something unknown";
if (e instanceof ResourceAccessException) {
var resourceAccessException = (ResourceAccessException) e;
if (e.getCause() instanceof ConnectTimeoutException) {
cause = "Connection timeout";
}
if (e.getCause() instanceof SocketTimeoutException) {
cause = "Read timeout";
}
} else if (e instanceof HttpServerErrorException) {
var httpServerErrorException = (HttpServerErrorException) e;
cause = "Server error";
} else if (e instanceof HttpClientErrorException) {
var httpClientErrorException = (HttpClientErrorException) e;
cause = "Client error";
} else if (e instanceof CallNotPermittedException) {
var callNotPermittedException = (CallNotPermittedException) e;
cause = "Open circuit breaker";
}
var message = String.format("%s caused fallback, caught exception %s",
cause, e.getMessage());
log.error(message); // application log entry
throw new MyRestException (message, e);
}
When I call this method person() the retry happens as maxAttempt configured. I expect my custom runtime MyRestException is caught on each retry and thrown on the last one (when maxAttempt is reached), so I wrap the call in the try-catch.
public List<Person> person() {
try {
return myRestService.person().getBody();
} catch (MyRestException ex) {
log.error("Here I am ready to log the issue into the database");
throw new ex;
}
}
Unfortunatelly, the retry seems to be ignored as the fallback encounters and rethrows the exception that is immediatelly caught with my try-catch instead of the Resilience4j-retry mechanism.
How to achieve the behavior when the maxAttempts is hit? Is there a way to define a specific fallback method for such case?

Why don't you catch and map exceptions to MyRestException inside of your Service methods, e.g. myRestService.person()?
It makes your configuration even simpler, because you only have to add MyRestException to the configuration of your RetryConfig and CircuitBreakerConfig.
Spring RestTemplate also has mechanisms to register a custom ResponseErrorHandler, if you don't want to add the boilerplate code to every Service method. -> https://www.baeldung.com/spring-rest-template-error-handling
I would not map CallNotPermittedException to MyRestException. You don't want to retry when the CircuitBreaker is open. Add CallNotPermittedException to the list of ignored exceptions in your RetryConfig.
I think you don't need the fallback mechanism at all. I thing mapping an exception to another exception is not a "fallback".

Spring Retry does not work on 2nd level of methods

#Retryable doesn't seem to be working on 2nd level of methods as in sphRemoteCall below. I see that a proxy is created but it is never retried on failures.
Once I moved #Retryable to the 1st level of methods like getSubscriberAccount, it's started working.
Example below:
#Service
public class SphIptvClient extends WebServiceGatewaySupport {
//Works over here
#Retryable(maxAttempts=3, backoff=#Backoff(delay=100))
public GetSubscriberAccountResponse getSubscriberAccount(String loginTocken, String billingServId) {
GetSubscriberAccountResponse response = (GetSubscriberAccountResponse) sphRemoteCall(sphIptvEndPoint, getSubAcc, "xxxxx");
return response;
}
/*
* Retryable is not working on the 2nd level methods in the bean.
* It works only with methods which are called directly from outside
* if there is 2nd level method, like this, Retryable is not working.
*/
//#Retryable
private Object sphRemoteCall(String uri, Object requestPayload, String soapAction) {
log.debug("Calling the sph for uri:{} and soapAction:{}", uri, soapAction);
return getWebServiceTemplate().marshalSendAndReceive(uri, requestPayload, new SoapActionCallback(soapAction));
}
}
#Configuration
#EnableRetry
public class SphClientConfig {
#Bean
public SphIptvClient sphIptvClient() {
SphIptvClient client = new SphIptvClient();
return client;
}
}

So this is a super late answer, but since I've just come here and confronted the same problem (again, after years ago wrestling with transactions) I'll furnish a little more fleshed out solution and hopefully someone will find it useful. Suffice to say that #M. Deinum's diagnosis is correct.
In the above case, and to paraphrase Understanding AOP proxies, any place where SphIptvClient gets autowired will be given a reference to a proxy which Spring Retry will create when #EnableRetry is handled:
"The #EnableRetry annotation creates proxies for #Retryable beans" - Declarative Retry - Spring Retry
Once getSubscriberAccount has been invoked and execution has passed through the proxy and into the #Service instance of the object, no reference to the proxy is known. As a result sphRemoteCall is called as if there were no #Retryable at all.
You could work with the framework by shuffling code around in such a way as to allow getSubscriberAccount to call a proxy-ed sphRemoteCall, which requires a new interface and class implementation.
For example:
public interface SphWebService {
Object sphRemoteCall(String uri, Object requestPayload, String soapAction);
}
#Component
public class SphWebServiceImpl implements SphWebService {
#Retryable
public Object sphRemoteCall(String uri, Object requestPayload, String soapAction) {
log.debug("Calling the sph for uri:{} and soapAction:{}", uri, soapAction);
return getWebServiceTemplate().marshalSendAndReceive(uri, requestPayload, new SoapActionCallback(soapAction));
}
}
#Service
public class SphIptvClient extends WebServiceGatewaySupport {
#Autowired
SphWebService sphWebService;
#Retryable(maxAttempts=3, backoff=#Backoff(delay=100))
public GetSubscriberAccountResponse getSubscriberAccount(String loginTocken, String billingServId) {
GetSubscriberAccountResponse response = (GetSubscriberAccountResponse) this.sphWebService.sphRemoteCall(sphIptvEndPoint, getSubAcc, "xxxxx");
return response;
}
}
#Configuration
#EnableRetry
public class SphClientConfig {
// the #Bean method was unnecessary and may cause confusion.
// #Service was already instantiating SphIptvClient behind the scenes.
}

#Retryable only works on the methods when called directly from other classes.
If you will try to invoke one method with #Retryable annotation from some other method of the same class, it will eventually not work.
// any call from this method to test method will not invoke the retry logic.
public void yetAnotherMethod() {
this.test();
}
// it will work
#Retryable(value = {RuntimeException.class}, backoff = #Backoff(delay = 1500))
public void test() {
System.out.println("Count: " + count++);
throw new RuntimeException("testing");
}
#Recover
public void recover() {
System.out.println("Exception occured.");
}
So, the output if test method is called, will be:
Count: 0
Count: 1
Count: 2
Exception occured.
But, if the yetAnotherMethod is called, output will be:
Count: 0
And a Runtime exception will be thrown.

Suppose you have a method which calls certain API - callAPI() and you want to implement retry logic over it, you can try use a do while, as it will execute only once, if successful.
Method to hit the external API
public int callAPI() {
return 1;
}
Method to implement retry logic
public int retrylogic() throws InterruptedException {
int retry = 0;
int status = -1;
boolean delay = false;
do {
// adding a delay, if you want some delay between successive retries
if (delay) {
Thread.sleep(2000);
}
// Call the actual method, and capture the response,
// and also catch any exception which occurs during the call.
// (Network down/ endpoint not avaliable
try {
status = callAPI();
}
catch (Exception e) {
System.out.println("Error occured");
status = -1;
}
finally {
switch (status) { //now based on error response or any exception you retry again
case HTTPStatus.OK:
System.out.println("OK");
return status;
default:
System.out.println("Unknown response code");
break;
}
retry++;
System.out.println("Failed retry " + retry + "/" + 3);
delay = true;
}
}while (retry < 3);
return status;
}

ThreadLocal and #Aspect annotation

I am using #Aspect to implement a retry logic(max_retries = 5) for database stale connection problems.In this Advice I have a ThreadLocal object which keep tracks of how many times logic has retried to get connection and it gets incremented whenever it cannot get connection so to avoid unlimited retries for stale connection issue, maximum number of retries is 5(constant).
But the problem I have is , in this #Aspect java class ThreadLocal never gets incremented and this is causing endlees loop in the code, which of course should not retry after maximun number of retries, but never reach that count and does not break out of while loop.
Please let me know if anybody had this problem with #Aspect and ThreadLcal object.
I will be happy to share the code.
private static ThreadLocal<Integer> retryCounter= new ThreadLocal<Integer>() {};
private static final String STALE_CONNECTION_EXCEPTION = "com.ibm.websphere.ce.cm.StaleConnectionException";
#Around("service")
public Object retryConnection(ProceedingJoinPoint pjp) throws Throwable {
if (staleConnectionException == null) {
return pjp.proceed();
}
Throwable exception = null;
retryCounter.set(new Integer(0));
while ( retryCounter.get() < MAX_TRIES) {
try {
return pjp.proceed();
}
catch (AppDataException he) {
exception = retry(he.getCause());
}
catch (NestedRuntimeException e) {
exception = retry(e);
}
}
if (exception != null) {
Logs.error("Stale connection exception occurred, no more retries left", this.getClass(), null);
logException(pjp, exception);
throw new AppDataException(exception);
}
return null;
}
private Throwable retry(Throwable e) throws Throwable {
if (e instanceof NestedRuntimeException && ((NestedRuntimeException)e).contains(staleConnectionException)) {
retryCounter.set(retryCounter.get()+1);
LogUtils.log("Stale connection exception occurred, retrying " + retryCounter.get() + " of " + MAX_TRIES, this.getClass());
return e;
}
else {
throw e;
}
}

As mentioned in the comments, not sure why you are using a thread local... but given that you are, what might be causing the infinite loop is recursive use of this aspect. Run it through a debugger or profile it to see if you are hitting the same aspect in a nested fashion.

To be honest, looking at your code, I think you would be better off not doing this at all, but rather just configure connection testing in your connection pool (assuming you are using one): http://pic.dhe.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.multiplatform.doc/info/ae/ae/tdat_pretestconn.html

Spring Optimistic Locking:How to retry transactional method till commit is successful

I use Spring 2.5 and Hibernate JPA implementation with Java and "container" managed Transactions.
I have a "after user commit" method that updates data in background and need to be committed regardless of ConcurrencyFailureException or StaleObjectStateException exception, because it will never be shown to client. In other words, need to make Optimistic Lock to Pessimistic. (Could happen if methods execution will take little bit longer and someone changed data in other transaction)
I read a a lot about idempotent stuff, retry if exception in search for DEFAULT_MAX_RETRIES or 6.2.7. Example or chapter 14.5. Retry. I also found in stackoverflow here and here.
I tried this:
public aspect RetryOnConcurrencyExceptionAspect {
private static final int DEFAULT_MAX_RETRIES = 20;
private int maxRetries = DEFAULT_MAX_RETRIES;
Object around(): execution( * * (..) ) && #annotation(RetryOnConcurrencyException) && #annotation(Transactional) {
int numAttempts = 0;
RuntimeException failureException = null;
do {
numAttempts++;
try {
return proceed();
}
catch( OptimisticLockingFailureException ex ) {
failureException = ex;
}
catch(ConcurrencyFailureException ex) {
failureException = ex;
}
catch( StaleObjectStateException ex) {
failureException = ex;
}
} while( numAttempts <= this.maxRetries );
throw failureException;
}
}
RetryOnConcurrencyException is my Annotation to mark methods that need to be retried, if a exception occurrs. Didn't work... I also tried several ways like SELECT ... FOR UPDATE, EntityManager.lock(...)
What is the best way to avoid stale data, dirty reads etc. such a strategy with Spring? Retry?, synchronized?, JPA lock?, isolation?, select ... for update? I could not get it to work and I really happy about any help.
Here is some pseudo code what I like to do:
void doSomething(itemId) {
select something into A;
select anotherthing into B;
// XXX
item = getItemFormDB( itemId ); // takes long for one user and for other concurrent user it could take less time
item.setA(A);
item.setB(B);
// YYYY
update item;
}
Between // XXX and // YYY another session could modify the item, then the StaleObjectStateException gets thrown.

Use Spring Retry to retry whole method when a version number or timestamp check failed (optimistic lock occurs).
Configuration
#Configuration
#EnableRetry
public class FooConfig {
...
}
Usage
#Retryable(StaleStateException.class)
#Transactional
public void doSomethingWithFoo(Long fooId){
// read your entity again before changes!
Foo foo = fooRepository.findOne(fooId);
foo.setStatus(REJECTED) // <- sample foo modification
} // commit on method end
Project configuration
Spring Boot application has defined valid spring-retry version, so only this is required:
<dependency>
<groupId>org.springframework.retry</groupId>
<artifactId>spring-retry</artifactId>
</dependency>

I got a solution but I think it's ugly. I catch all RuntimeException and it only works for new transactions. Do you know how to make it better? Do you see any problems?
First, I made an Annotation:
#Target({ElementType.METHOD})
#Retention(RetentionPolicy.RUNTIME)
public #interface RetryingTransaction {
int repeatCount() default 20;
}
Then I made a interceptor like this:
public class RetryingTransactionInterceptor implements Ordered {
private static final int DEFAULT_MAX_RETRIES = 20;
private int maxRetries = DEFAULT_MAX_RETRIES;
private int order = 1;
#Resource
private PlatformTransactionManager transactionManager;
public void setMaxRetries(int maxRetries) {
this.maxRetries = maxRetries;
}
public int getOrder() {
return this.order;
}
public void setOrder(int order) {
this.order = order;
}
public Object retryOperation(ProceedingJoinPoint pjp) throws Throwable {
int numAttempts = 0;
Exception failureException = null;
do {
numAttempts++;
try {
DefaultTransactionDefinition def = new DefaultTransactionDefinition();
def.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
TransactionStatus status = transactionManager.getTransaction(def);
Object obj = pjp.proceed();
transactionManager.commit(status);
return obj;
}
catch( RuntimeException re ) {
failureException = re;
}
} while( numAttempts <= this.maxRetries );
throw failureException;
}
}
Spring applicationConfig.xml:
<tx:annotation-driven transaction-manager="transactionManager" order="10" />
<bean id="transactionManager" class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="transactionSynchronizationName">
<value>SYNCHRONIZATION_ALWAYS</value>
</property>
</bean>
<bean id="retryingTransactionInterceptor" class="com.x.y.z.transaction.RetryingTransactionInterceptor">
<property name="order" value="1" />
</bean>
<aop:config>
<aop:aspect id="retryingTransactionAspect" ref="retryingTransactionInterceptor">
<aop:pointcut
id="servicesWithRetryingTransactionAnnotation"
expression="execution( * com.x.y.z.service..*.*(..) ) and #annotation(com.x.y.z.annotation.RetryingTransaction)"/>
<aop:around method="retryOperation" pointcut-ref="servicesWithRetryingTransactionAnnotation"/>
</aop:aspect>
</aop:config>
And a method annotated like this:
#RetryingTransaction
public Entity doSomethingInBackground(params)...

We have this and what we do is:
Flush the session (to make sure the upcoming update will be the only one queued)
Load the instance
Do the change
On StaleObjectStateException, clear the action queue
((EventSource) session).getActionQueue().clear()
and retry from #2
We have a retry counter to re-throw the exception in the end.
NOTE: This is not an officially supported method (Hibernate clearly states that a session which has thrown an exception should be discarded and not re-used), but it's a known work-around (with the limitation that you can't selectively remove the update action, but must clear the whole queue).

Throwing out another option here: BoneCP (http://jolbox.com) has support to automatically retry transactions upon failure (including when DB goes down, network fails, etc).