Develop Reference

Handle exception after reaching max attempts in resilience4j-retry using Spring Boot

I have a scenario I want to log each retry attempt and when the last one fails (i.e. maxAttempts reached) a exception is thrown and let's say an entry to a database is created.
I try to achieve this using Resilience4j-retry with Spring Boot, therefore I use application.yml and annotations.
#Retry(name = "default", fallbackMethod="fallback")
#CircuitBreaker(name = "default", fallbackMethod="fallback")
public ResponseEntity<List<Person>> person() {
return restTemplate.exchange(...); // let's say this always throws 500
}
The fallback logs the cause of the exception into an application log.
public ResponseEntity<?> fallback(Exception e) {
var status = HttpStatus.INTERNAL_SERVER_ERROR;
var cause = "Something unknown";
if (e instanceof ResourceAccessException) {
var resourceAccessException = (ResourceAccessException) e;
if (e.getCause() instanceof ConnectTimeoutException) {
cause = "Connection timeout";
}
if (e.getCause() instanceof SocketTimeoutException) {
cause = "Read timeout";
}
} else if (e instanceof HttpServerErrorException) {
var httpServerErrorException = (HttpServerErrorException) e;
cause = "Server error";
} else if (e instanceof HttpClientErrorException) {
var httpClientErrorException = (HttpClientErrorException) e;
cause = "Client error";
} else if (e instanceof CallNotPermittedException) {
var callNotPermittedException = (CallNotPermittedException) e;
cause = "Open circuit breaker";
}
var message = String.format("%s caused fallback, caught exception %s",
cause, e.getMessage());
log.error(message); // application log entry
throw new MyRestException (message, e);
}
When I call this method person() the retry happens as maxAttempt configured. I expect my custom runtime MyRestException is caught on each retry and thrown on the last one (when maxAttempt is reached), so I wrap the call in the try-catch.
public List<Person> person() {
try {
return myRestService.person().getBody();
} catch (MyRestException ex) {
log.error("Here I am ready to log the issue into the database");
throw new ex;
}
}
Unfortunatelly, the retry seems to be ignored as the fallback encounters and rethrows the exception that is immediatelly caught with my try-catch instead of the Resilience4j-retry mechanism.
How to achieve the behavior when the maxAttempts is hit? Is there a way to define a specific fallback method for such case?

Why don't you catch and map exceptions to MyRestException inside of your Service methods, e.g. myRestService.person()?
It makes your configuration even simpler, because you only have to add MyRestException to the configuration of your RetryConfig and CircuitBreakerConfig.
Spring RestTemplate also has mechanisms to register a custom ResponseErrorHandler, if you don't want to add the boilerplate code to every Service method. -> https://www.baeldung.com/spring-rest-template-error-handling
I would not map CallNotPermittedException to MyRestException. You don't want to retry when the CircuitBreaker is open. Add CallNotPermittedException to the list of ignored exceptions in your RetryConfig.
I think you don't need the fallback mechanism at all. I thing mapping an exception to another exception is not a "fallback".

#Transactional method insert value on exception and multithread wildfly CDI

I have a method in CDI bean which is transactional, on error it creates an entry in database with the exception message. This method can be called by RESTendpoint and in multithread way.
We have a SQL constraint to avoid duplicity in database
#Transactional
public RegistrationRuleStatus performCheck(RegistrationRule rule, User user) {
try {
//check if rule is dependent of other rules and if all proved, perform check
List<RegistrationRule> rules = rule.getRuleParentDependencies();
boolean parentDependenciesAreProved = true;
if (!CollectionUtils.isEmpty(rules)) {
parentDependenciesAreProved = ruleDao.areParentDependenciesProved(rule,user.getId());
}
if (parentDependenciesAreProved) {
Object service = CDI.current().select(Object.class, new NamedAnnotation(rule.getProvider().name())).get();
Method method = service.getClass().getMethod(rule.getProviderType().getMethod(), Long.class, RegistrationRule.class);
return (RegistrationRuleStatus) method.invoke(service, user.getId(), rule);
} else {
RegistrationRuleStatus status = statusDao.getStatusByUserAndRule(user, rule);
if (status == null) {
status = new RegistrationRuleStatus(user, rule, RegistrationActionStatus.START, new Date());
statusDao.create(status);
}
return status;
}
} catch (Exception e) {
LOGGER.error("could not perform check {} for provider {}", rule.getProviderType().name(), rule.getProvider().name(), e.getCause()!=null?e.getCause():e);
return statusDao.createErrorStatus(user,rule,e.getCause()!=null?e.getCause().getMessage():e.getMessage());
}
}
create Error method:
#Transactional
public RegistrationRuleStatus createErrorStatus(User user, RegistrationRule rule, String message) {
RegistrationRuleStatus status = getStatusByUserAndRule(user, rule);
if (status == null) {
status = new RegistrationRuleStatus(user, rule, RegistrationActionStatus.ERROR, new Date());
status.setErrorCode(CommonPropertyResolver.getMicroServiceErrorCode());
status.setErrorMessage(message);
create(status);
}else {
status.setStatus(RegistrationActionStatus.ERROR);
status.setStatusDate(new Date());
status.setErrorCode(CommonPropertyResolver.getMicroServiceErrorCode());
status.setErrorMessage(message);
update(status);
}
return status;
}
the problem is method is called twice at same time and the error recorded is DuplicateException but we don't want it. We verify at the beginning if object already exists, but I think it is called at exactly same time.
JAVA8/wildfly/CDI/JPA/eclipselink
Any idea?

I'd suggest you to consider following approaches:
1) Implement retry logic. Catch exception, analyze it. If it indicates an unexpected duplicate (like you described), then don't consider it as an error and just repeat the method call. Now your code will work differently: It will notice that a record already exists and will not create a duplicate.
2) Use isolation level SERIALIZABLE. Then within a single transaction your will "see" a consistent behaviour: If select operation hasn't found a particular record, then till the end of this transaction no other transaction will insert such record and there will be no exception related to duplicates. But the price is that the whole table will be locked for each such transaction. This can degrade the application performance essentially.

Lock wait timeout exceeded with Hibernate and MySQL (using play framework)

In my web application I'm using Stateless sessions with Hibernate to have better performances on my inserts and updates.
It was working fine with H2 database (the one used in play framework in dev mode).
But when I test it with MySQL I get the following exception :
ERROR ~ Lock wait timeout exceeded; try restarting transaction
ERROR ~ HHH000315: Exception executing batch [Lock wait timeout exceeded; try restarting transaction]
Here is the code :
public static void update() {
Session session = (Session) JPA.em().getDelegate();
StatelessSession stateless = this.session.getSessionFactory().openStatelessSession();
try {
stateless.beginTransaction();
// Fetch all products
{
List<ProductType> list = ProductType.retrieveAllWithHistory();
for (ProductType pt : list) {
updatePrice(pt, stateless);
}
}
// Fetch all raw materials
{
List<RawMaterialType> list = RawMaterialType.retrieveAllWithHistory();
for (RawMaterialType rm : list) {
updatePrice(rm, stateless);
}
}
} catch (Exception ex) {
play.Logger.error(ex.getMessage());
ExceptionLog.log(ex, Thread.currentThread());
} finally {
stateless.getTransaction().commit();
stateless.close();
}
}
private static void updatePrice(ProductType pt, StatelessSession stateless) {
pt.priceDelta = computeDelta();
pt.unitPrice = computePrice();
stateless.update(pt);
PriceHistory ph = new PriceHistory(pt, price);
stateless.insert(ph);
}
private static void updatePrice(RawMaterialType rm, StatelessSession stateless) {
rm.priceDelta = computeDelta();
rm.unitPrice = computePrice();
stateless.update(rm);
PriceHistory ph = new GoodPriceHistory(rm, price);
stateless.insert(ph);
}
In this example I have 3 simple Entities (ProductType, RawMaterialType and PriceHistory).
computeDelta and computePrice are just algorithm functions with no DB stuff.
retrieveAllWithHistory functions are functions that fetch some data from the database using Play framework model functions.
So, this code retrieves some data, edit some, create new one and finally save everything.
Why have I a lock exception with MySQL and no exception with H2 ?

I'm not sure why you have a commit in a finally block. Give this structure a try:
try {
factory.getCurrentSession().beginTransaction();
factory.getCurrentSession().getTransaction().commit();
} catch (RuntimeException e) {
factory.getCurrentSession().getTransaction().rollback();
throw e; // or display error message
}
Also, it might be helpful for you to check this documentation.

ThreadLocal and #Aspect annotation

I am using #Aspect to implement a retry logic(max_retries = 5) for database stale connection problems.In this Advice I have a ThreadLocal object which keep tracks of how many times logic has retried to get connection and it gets incremented whenever it cannot get connection so to avoid unlimited retries for stale connection issue, maximum number of retries is 5(constant).
But the problem I have is , in this #Aspect java class ThreadLocal never gets incremented and this is causing endlees loop in the code, which of course should not retry after maximun number of retries, but never reach that count and does not break out of while loop.
Please let me know if anybody had this problem with #Aspect and ThreadLcal object.
I will be happy to share the code.
private static ThreadLocal<Integer> retryCounter= new ThreadLocal<Integer>() {};
private static final String STALE_CONNECTION_EXCEPTION = "com.ibm.websphere.ce.cm.StaleConnectionException";
#Around("service")
public Object retryConnection(ProceedingJoinPoint pjp) throws Throwable {
if (staleConnectionException == null) {
return pjp.proceed();
}
Throwable exception = null;
retryCounter.set(new Integer(0));
while ( retryCounter.get() < MAX_TRIES) {
try {
return pjp.proceed();
}
catch (AppDataException he) {
exception = retry(he.getCause());
}
catch (NestedRuntimeException e) {
exception = retry(e);
}
}
if (exception != null) {
Logs.error("Stale connection exception occurred, no more retries left", this.getClass(), null);
logException(pjp, exception);
throw new AppDataException(exception);
}
return null;
}
private Throwable retry(Throwable e) throws Throwable {
if (e instanceof NestedRuntimeException && ((NestedRuntimeException)e).contains(staleConnectionException)) {
retryCounter.set(retryCounter.get()+1);
LogUtils.log("Stale connection exception occurred, retrying " + retryCounter.get() + " of " + MAX_TRIES, this.getClass());
return e;
}
else {
throw e;
}
}

As mentioned in the comments, not sure why you are using a thread local... but given that you are, what might be causing the infinite loop is recursive use of this aspect. Run it through a debugger or profile it to see if you are hitting the same aspect in a nested fashion.

To be honest, looking at your code, I think you would be better off not doing this at all, but rather just configure connection testing in your connection pool (assuming you are using one): http://pic.dhe.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.multiplatform.doc/info/ae/ae/tdat_pretestconn.html

Hibernate 4.1.9 (latest final build) reporting `nested transactions not supported`

I am getting a
org.hibernate.TransactionException: nested transactions not supported
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:152)
at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1395)
at com.mcruiseon.server.hibernate.ReadOnlyOperations.flush(ReadOnlyOperations.java:118)
Code that throws that exception. I am calling flush from a thread that runs infinite until there is data to flush.
public void flush(Object dataStore) throws DidNotSaveRequestSomeRandomError {
Transaction txD;
Session session;
session = currentSession();
// Below Line 118
txD = session.beginTransaction();
txD.begin() ;
session.saveOrUpdate(dataStore);
try {
txD.commit();
while(!txD.wasCommitted()) ;
} catch (ConstraintViolationException e) {
txD.rollback() ;
throw new DidNotSaveRequestSomeRandomError(dataStore, feedbackManager);
} catch (TransactionException e) {
txD.rollback() ;
} finally {
// session.flush();
txD = null;
session.close();
}
// mySession.clear();
}
Edit :
I am calling flush in a independent thread as datastore list contains data. From what I see its a sync operation call to flush, so ideally flush should not return until transaction is complete. I would like it that way is the least I want to expect. Since its a independent thread doing its job, all I care about it flush being a sync operation. Now my question is, is txD.commit a async operation ? Does it return before that transaction has a chance to finish. If yes, is there a way to get commit to "Wait" until the transaction completes ?
public void run() {
Object dataStore = null;
while (true) {
try {
synchronized (flushQ) {
if (flushQ.isEmpty())
flushQ.wait();
if (flushQ.isEmpty()) {
continue;
}
dataStore = flushQ.removeFirst();
if (dataStore == null) {
continue;
}
}
try {
flush(dataStore);
} catch (DidNotSaveRequestSomeRandomError e) {
e.printStackTrace();
log.fatal(e);
}
} catch (HibernateException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
Edit 2 : Added while(!txD.wasCommitted()) ; (in code above), still I get that freaking nested transactions not supported. Infact due to this exception a record is not being written to by table too. Is there something to do with the type of table ? I have INNODB for all my tables?

Finally got the nested transaction not supported error fixed. Changes made to code are
if (session.getTransaction() != null
&& session.getTransaction().isActive()) {
txD = session.getTransaction();
} else {
txD = session.beginTransaction();
}
//txD = session.beginTransaction();
// txD.begin() ;
session.saveOrUpdate(dataStore);
try {
txD.commit();
while (!txD.wasCommitted())
;
}
Credits of above code also to Venkat. I did not find HbTransaction, so just used getTransaction and beginTransaction. It worked.
I also made changes in the hibernate properties due to advice on here. I added these lines to the hibernate.properties. This alone did not solve the issue. But I am leaving it there.
hsqldb.write_delay_millis=0
shutdown=true

You probably already began a transaction before calling this method.
Either this should be part of the enclosing transaction, and you should thus not start another one; or it shouldn't be part of the enclosing transaction, and you should thus open a new session and a new transaction rather than using the current session.