I am using #Aspect to implement a retry logic(max_retries = 5) for database stale connection problems.In this Advice I have a ThreadLocal object which keep tracks of how many times logic has retried to get connection and it gets incremented whenever it cannot get connection so to avoid unlimited retries for stale connection issue, maximum number of retries is 5(constant).
But the problem I have is , in this #Aspect java class ThreadLocal never gets incremented and this is causing endlees loop in the code, which of course should not retry after maximun number of retries, but never reach that count and does not break out of while loop.
Please let me know if anybody had this problem with #Aspect and ThreadLcal object.
I will be happy to share the code.
private static ThreadLocal<Integer> retryCounter= new ThreadLocal<Integer>() {};
private static final String STALE_CONNECTION_EXCEPTION = "com.ibm.websphere.ce.cm.StaleConnectionException";
#Around("service")
public Object retryConnection(ProceedingJoinPoint pjp) throws Throwable {
if (staleConnectionException == null) {
return pjp.proceed();
}
Throwable exception = null;
retryCounter.set(new Integer(0));
while ( retryCounter.get() < MAX_TRIES) {
try {
return pjp.proceed();
}
catch (AppDataException he) {
exception = retry(he.getCause());
}
catch (NestedRuntimeException e) {
exception = retry(e);
}
}
if (exception != null) {
Logs.error("Stale connection exception occurred, no more retries left", this.getClass(), null);
logException(pjp, exception);
throw new AppDataException(exception);
}
return null;
}
private Throwable retry(Throwable e) throws Throwable {
if (e instanceof NestedRuntimeException && ((NestedRuntimeException)e).contains(staleConnectionException)) {
retryCounter.set(retryCounter.get()+1);
LogUtils.log("Stale connection exception occurred, retrying " + retryCounter.get() + " of " + MAX_TRIES, this.getClass());
return e;
}
else {
throw e;
}
}
As mentioned in the comments, not sure why you are using a thread local... but given that you are, what might be causing the infinite loop is recursive use of this aspect. Run it through a debugger or profile it to see if you are hitting the same aspect in a nested fashion.
To be honest, looking at your code, I think you would be better off not doing this at all, but rather just configure connection testing in your connection pool (assuming you are using one): http://pic.dhe.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.multiplatform.doc/info/ae/ae/tdat_pretestconn.html
Related
I have a Spring Boot application code similar to one given below, it tries to save a list of entities in MongoDB, but I am getting this error:
Exception in thread "Thread-20" org.springframework.data.mongodb.UncategorizedMongoDbException: Command failed with error 112 (WriteConflict): 'WriteConflict error: this operation conflicted with another operation. Please retry your operation or multi-document transaction.' on server 127.X.X.X:XXXXX. The full response is {"errorLabels": ["TransientTransactionError"]
Caused by: com.mongodb.MongoCommandException: Command failed with error 112 (WriteConflict): 'WriteConflict error: this operation conflicted with another operation. Please retry your operation or multi-document transaction.' on server 127.X.X.X:XXXXX. The full response is {"errorLabels": ["TransientTransactionError"]
So, what should I do to avoid this error/exception?
Code:
#Transactional
public synchronized void myMethod(List<MyEntity> myEntities) {
saveEntities(myEntities);
}
public void saveEntities(List<MyEntity> myEntities) {
myRepository.saveAll(myEntities);
}
Java mongo driver (mongo-driver-sync-4.4.2) supports retry for WriteConflicts error:
ClientSessionImpl.withTransaction(final TransactionBody<T> transactionBody, final TransactionOptions options)
outer:
while (true) {
T retVal;
try {
startTransaction(options);
retVal = transactionBody.execute();
} catch (RuntimeException e) {
if (transactionState == TransactionState.IN) {
abortTransaction();
}
if (e instanceof MongoException) {
if (((MongoException) e).hasErrorLabel(TRANSIENT_TRANSACTION_ERROR_LABEL)
&& ClientSessionClock.INSTANCE.now() - startTime < MAX_RETRY_TIME_LIMIT_MS) {
continue;
}
}
throw e;
}
if (transactionState == TransactionState.IN) {
while (true) {
try {
commitTransaction();
break;
} catch (MongoException e) {
clearTransactionContextOnError(e);
if (ClientSessionClock.INSTANCE.now() - startTime < MAX_RETRY_TIME_LIMIT_MS) {
applyMajorityWriteConcernToTransactionOptions();
if (!(e instanceof MongoExecutionTimeoutException)
&& e.hasErrorLabel(UNKNOWN_TRANSACTION_COMMIT_RESULT_LABEL)) {
continue;
} else if (e.hasErrorLabel(TRANSIENT_TRANSACTION_ERROR_LABEL)) {
continue outer;
}
}
throw e;
}
}
}
return retVal;
}
Notice if (e.hasErrorLabel(TRANSIENT_TRANSACTION_ERROR_LABEL) redirects logic to outer which is to start the whole block again.
However, Spring Boot doesn't make use of withTransaction to start a transaction, at least as far as I can see from using Transactional annotation.
The only way is to retry at application level. One way is Retryable
#Retryable(
value = UncategorizedMongoDbException.class,
exceptionExpression = "#{message.contains('WriteConflict error')}",
maxAttempts = 12, backoff = #Backoff(delay = 500))
Make sure to apply it on methods that executes database logic only, otherwise, it can cause unwanted behaviour such as repeat certain non-database operation multiple times when WriteConflicts occurred
I have a scenario I want to log each retry attempt and when the last one fails (i.e. maxAttempts reached) a exception is thrown and let's say an entry to a database is created.
I try to achieve this using Resilience4j-retry with Spring Boot, therefore I use application.yml and annotations.
#Retry(name = "default", fallbackMethod="fallback")
#CircuitBreaker(name = "default", fallbackMethod="fallback")
public ResponseEntity<List<Person>> person() {
return restTemplate.exchange(...); // let's say this always throws 500
}
The fallback logs the cause of the exception into an application log.
public ResponseEntity<?> fallback(Exception e) {
var status = HttpStatus.INTERNAL_SERVER_ERROR;
var cause = "Something unknown";
if (e instanceof ResourceAccessException) {
var resourceAccessException = (ResourceAccessException) e;
if (e.getCause() instanceof ConnectTimeoutException) {
cause = "Connection timeout";
}
if (e.getCause() instanceof SocketTimeoutException) {
cause = "Read timeout";
}
} else if (e instanceof HttpServerErrorException) {
var httpServerErrorException = (HttpServerErrorException) e;
cause = "Server error";
} else if (e instanceof HttpClientErrorException) {
var httpClientErrorException = (HttpClientErrorException) e;
cause = "Client error";
} else if (e instanceof CallNotPermittedException) {
var callNotPermittedException = (CallNotPermittedException) e;
cause = "Open circuit breaker";
}
var message = String.format("%s caused fallback, caught exception %s",
cause, e.getMessage());
log.error(message); // application log entry
throw new MyRestException (message, e);
}
When I call this method person() the retry happens as maxAttempt configured. I expect my custom runtime MyRestException is caught on each retry and thrown on the last one (when maxAttempt is reached), so I wrap the call in the try-catch.
public List<Person> person() {
try {
return myRestService.person().getBody();
} catch (MyRestException ex) {
log.error("Here I am ready to log the issue into the database");
throw new ex;
}
}
Unfortunatelly, the retry seems to be ignored as the fallback encounters and rethrows the exception that is immediatelly caught with my try-catch instead of the Resilience4j-retry mechanism.
How to achieve the behavior when the maxAttempts is hit? Is there a way to define a specific fallback method for such case?
Why don't you catch and map exceptions to MyRestException inside of your Service methods, e.g. myRestService.person()?
It makes your configuration even simpler, because you only have to add MyRestException to the configuration of your RetryConfig and CircuitBreakerConfig.
Spring RestTemplate also has mechanisms to register a custom ResponseErrorHandler, if you don't want to add the boilerplate code to every Service method. -> https://www.baeldung.com/spring-rest-template-error-handling
I would not map CallNotPermittedException to MyRestException. You don't want to retry when the CircuitBreaker is open. Add CallNotPermittedException to the list of ignored exceptions in your RetryConfig.
I think you don't need the fallback mechanism at all. I thing mapping an exception to another exception is not a "fallback".
I have a MessageBean which reads from a Queue we'll name MainQ.
If the execution of the onMessage code throws a user-based Exception with a type we'll name UserException I want to catch this and put this message on a separate Queue named UserErrorQ.
If the exception is not of this type, the Exception is thrown on to be handled by the DMQ.
Here is my issue:
in my catch block I attempt, through a ErrorQueueHandler, to put this new message on the UserErrorQ. This results in an error when I attempt to connect to the connectionFactory to send the message to the UserErrorQ.
Apparently creating a new connection to a QueueConnectionFactory(javax.jms.ConnectionFactory) is causing problems
Error:
com.sun.messaging.jms.JMSException: MQRA:DCF:allocation failure:createConnection:Error in allocating a connection. Cause: javax.transaction.RollbackException
at com.sun.messaging.jms.ra.DirectConnectionFactory._allocateConnection(DirectConnectionFactory.java:548)
at com.sun.messaging.jms.ra.DirectConnectionFactory.createConnection(DirectConnectionFactory.java:265)
at com.sun.messaging.jms.ra.DirectConnectionFactory.createConnection(DirectConnectionFactory.java:244)`
MessageBean:
#TransactionAttribute(TransactionAttributeType.REQUIRED)
public void onMessage(Message message) {
try{
.
.
}catch(Exception e){
if(isUserExceptionWrappedInException(e){
errorQueueHandler.sendToErrorQueue(message);
}
}
}
private boolean isUserExceptionWrappedInException(Throwable t) {
if (t == null)
return false;
else if (t instanceof UserException)
return true;
else
return isUserExceptionWrappedInException(t.getCause());
}
ErrorQueueHandler:
public void sendToErrorQueue(Message message) {
try {
createConnection();
send((TextMessage)message);
} finally {
closeConnection();
}
}
private void createConnection() throws Exception {
try {
connection = connectionfactory.createConnection();
connection.start();
} catch (JMSException e) {
String msg = "Error while attempting to initialize connection to jms destination " + ERROR_QUEUE;
throw new OperationalException(msg, e, OperationalExceptionType.APPLIKASJONSTJENER);
}
}
As mentioned, the error occurs when attempting to make the connection. Anyone have a fix for this?
So, I have figured out the answer to my own question.
The reason for the connectionException was that the ErrorQueueHandler was not an EJB, but rather injected via CDI. There are no new instantiations allowed within a rollback state because the container discards the bean instance, which is why it failed. My REQUIRES_NEW annotation was also ignored as this belongs to the javax api, which will not affect a CDI injected bean.
Here are a few things to note:
Make sure the EJB has either no constructors, or public ones. The modifiers are important as the container needs these to be correct for it to instantiate the EJB.
There are a few problems with this approach.
As I am attempting to write the message to a separate error queue instead of the DMQ, I will have to consume the message and not throw the error on afterwards. Because the MDB is in a rollback state, the JMS spec clearly states that this will cause the message to be redelivered. What you will experience is that after writing to you custom errorQueue, the message will bounce right back to the queue and you now have an infinite loop.
Luckily i also have a solution:
The main issue here is controlling your transactions. For this scenario, i need 3 transactions:
One transaction for the MDB so that it is able to acknowledge the message event though i have a RuntimeException.
One transaction for the logic of the onMessage method so that i am able to do a rollback when i get an exception, but also still be able to write to the ErrorQueue.
One transaction for connecting and writing to the ErrorQueue while in a rollback state.
Code:
MessageBean:
#EJB
QueueService queueService;
#TransactionAttribute(TransactionAttributeType.REQUIRED)
public void onMessage(Message message) {
try{
queueService.processMessageInNewTrasaction(message);
}catch(Exception e){
throw e;
}
}
QueueService:
import javax.jms.Message;
#TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
#Stateless
public class QueueService {
#EJB
ErrorQueueHandler errorQueueHandler;
public void processMessageInNewTransaction(Message message){
try {
.
.
} catch(Exception e) {
if(isUserExceptionWrappedInException(e)
errorQueueHandler.sendToErrorQueue(message);
}
}
private boolean isUserExceptionWrappedInException(Throwable t) {
if (t == null)
return false;
else if (t instanceof UserException)
return true;
else
return isUserExceptionWrappedInException(t.getCause());
}
}
ErrorQueueHandler:
#TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
#Stateless
public class ErrorQueueHandler{
public void sendToErrorQueue(Message message){
.
.
}
}
useful resource: http://weblogic-wonders.com/weblogic/2011/01/10/working-with-jms-and-the-standard-issues-in-jms/
Is the following code considered a bad practice? Do you think it can be done otherwise?
The goal is to always update the status, either with success (i.e invocation to service.invoke(id);returns normally ) or with failure...
#Autowired
private Service service;
public void onMessage(Message message) {
String id = null;
String status = "FAILED";
try {
id = ((TextMessage) message).getText();
status = service.invoke(id); //can throw unchecked exception
} catch (final JMSException e) {
throw new RuntimeException(e);
} finally {
if (StringUtils.isNumeric(id)) {
service.update(id, status);
}
}
}
It depends on your Use-case, whether you have to perform a step or not based on previous step. Using finally may execute your second step regardless what exception you may receive.
I would recommend having the second step outside try...catch block so that you'll update only when you have got any exception you've Expected and continue to your second step, else, your method will throw and exit.
i think you should not use implementation of message listener , you should wire them independent of spring tech . just pojo based . use <jms:listener-container > with <jms:listener>
so i am using a filter to catch servlet exception (because we are using a mix of jsf/plain servlets)
when catching the ServletException and calling printstacktrace most of the information is lost.
the "true" root exception seems to be hidden behind the "funny" expression
((ServletException) e.getRootCause().getCause()).getRootCause().getCause().getCause().getCause()
this is clearly not the way to do it.
is the an easy way to print the "full" information of such an exception.
can someone explain me why the exception is wrapped this way?
Take a look at the ExceptionUtils class from commons-lang. It contains several useful methods for printing the entire chain of exceptions.
after i had a look at ExceptionUtils, this solved the problem!
final StringWriter stacktrace = new StringWriter();
ExceptionUtils.printRootCauseStackTrace(throwable,new PrintWriter(stacktrace));
msg.append(stacktrace.getBuffer());
this prints out the full stacktrace with every piece of information that is relevant.
That is called exception chaining. By wrapping an exception in a different exception you can let exceptions bubble up the stack without having your main application classes to worry about some low-level exceptions.
Example:
public void doStuff() throws StuffException {
try {
doDatabaseStuff();
} catch (DatabaseException de1) {
throw new StuffException("Could not do stuff in the database.", de1);
}
}
This way your application only has to handle StuffException but it can get to the underlying DatabaseException if it really needs to.
To get to the bottom-most (and all other) exception(s) of an exception you caught you can iterator over its root causes:
...
} catch (SomeException se1) {
Throwable t = se1;
logger.log(Level.WARNING, "Top exception", se1);
while (t.getCause() != null) {
t = t.getCause();
logger.log(Level.WARNING, "Nested exception", t);
}
// now t contains the root cause
}
Exception chaining for ServletException is tricky. Depending on the web server implementation and web development framework in use, at runtime the chain may use cause and/or rootCause. This link explains it very well. To complicate things, I've seen exceptions where the cause points to the exception itself.
Here's a recursive method we have used that covers all bases for ServletExceptions:
public static Throwable getDeepCause(Throwable ex) {
if (ex == null) {
return ex;
}
Throwable cause;
if (ex instanceof ServletException) {
cause = ((ServletException) ex).getRootCause();
if (cause == null) {
cause = ex.getCause();
}
} else {
cause = ex.getCause();
}
if (cause != null && cause != ex) {
return getDeepCause(cause);
} else {
// stop condition - reached the end of the exception chain
return ex;
}
}