I have used spring declarative retry in my project like
#Service
class Service {
#Async #Retryable(maxAttempts=12, backoff=#Backoff(delay=100, maxDelay=500))
public service() {
// ... do something
}
}
Now, I have two questions.
Is it fine to use retry with async, I don't have any issue, just
want to be sure.
The second requirement is, if the process fails I want to log it to log file including the number of remaining retries. So, is there a way to pass, or obtain the number of remaining retries from inside the method?
There is no way around using the annotations, #Recover annotated method executes only after the last failed retry, not after each one failing.
Refer to this github documentation
An excerpt from the link above- "Call the "service" method and if it fails with a RemoteAccessException then it will retry (up to three times by default), and then execute the "recover" method if unsuccessful."
Even with using RetryTemplate the Retry callback is called only after all retries are exhausted.
Another excerpt form the same link- "When a retry is exhausted the RetryOperations can pass control to a different callback, the RecoveryCallback. To use this feature clients just pass in the callbacks together to the same method"
You should use the #Recover annotation to perform an action on each fail and keep a count inside your object outside of the methods. Make sure no other methods interact with this counter. Here is the basic premise:
#Service
class Service {
private int attemptsLeft=12;
#Retryable(maxAttempts=12, backoff=#Backoff(delay=100, maxDelay=500))
public service() {
// ... do something that throws a KnownException you create to catch later on.
}
#Recover
public void connectionException(KnownException e) {
this.attemptsLeft = this.attemptsLeft-1; //decrease your failure counter
Logger.warn("Retry attempts left:{}",attemptsLeft);
}
}
If you don't want a member variable tracking count, you might need to ditch the annotations and declare the RetryTemplate to get access to the context, with the getRetryCount() method.
public String serviceWithRetry() {
RetryTemplate retryTemplate = new RetryTemplate();
final SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
retryPolicy.setMaxAttempts(12);
retryTemplate.setRetryPolicy(retryPolicy);
FixedBackOffPolicy backOffPolicy = new FixedBackOffPolicy();
backOffPolicy.setInterval(100L);
retryTemplate.setBackOffPolicy(backOffPolicy);
retryTemplate.execute(new RetryCallback<Void, RuntimeException>()
return retryTemplate.execute(new RetryCallback<Void, RuntimeException>() {
#Override
public void doWithRetry(RetryContext context) {
LOG.info("Retry of connection count: {}", context.getRetryCount());
return //something with your connection logic
}
});
}
Related
I have two micro-services, order-service, and inventory-service. The order-service makes a call to inventory-service to check if ordered items are in stock. An order is placed only if all the items in the order request are in stock. If the inventory-service is down or slow, the circuit breaker part triggers. The case of an order not being placed because of a missing item is not a failure case for the circuit breaker. This works well as intended only until the circuit has never switched from closed state to half open state.
What I mean by that is if more than 5 consecutive orders cannot be placed because of a missing item, the circuit does not switch to open state. This is as expected. If the inventory-service is brought down, 3 more failed requests cause the circuit to move to open and subsequently to half open state. This also is as expected. However when the inventory-service comes up again, and the only requests that are made are requests containing one or more items not in stock, the circuit remains in half_open state continuously. This is not ok. A missing item in an order is a success case and should increment the successful buffered call count, but it doesn't. Looking at the actuator info, it looks like these calls are not counted either as failure or as success cases.
What am I doing wrong.
Note -- if I make sufficient number of calls where order gets placed, then the circuit switches to closed again. That's ok but shouldn't the case of ignored exception count as a success case even if the only calls that are made are those with one or missing items.
Following are the properties of my circuit breaker in the calling microservice.
resilience4j.circuitbreaker.instances.inventory.register-health-indicator=true
resilience4j.circuitbreaker.instances.inventory.event-consumer-buffer-size=10
resilience4j.circuitbreaker.instances.inventory.sliding-window-type=COUNT_BASED
resilience4j.circuitbreaker.instances.inventory.sliding-window-size=5
resilience4j.circuitbreaker.instances.inventory.failure-rate-threshold=50
resilience4j.circuitbreaker.instances.inventory.wait-duration-in-open-state=5s
resilience4j.circuitbreaker.instances.inventory.permitted-number-of-calls-in-half-open-state=3
resilience4j.circuitbreaker.instances.inventory.automatic-transition-from-open-to-half-open-enabled=true
resilience4j.circuitbreaker.instances.inventory.ignore-exceptions=com.mayrevision.orderservice.exception.OrderItemNotFoundException
I have a custom exception handler for OrderItemNotFoundException.
#ControllerAdvice
#ResponseStatus
public class OrderItemNotFoundExceptionHandler extends ResponseEntityExceptionHandler {
#ExceptionHandler(OrderItemNotFoundException.class)
public ResponseEntity<ErrorResponse> getErrorMessage(OrderItemNotFoundException exception) {
ErrorResponse response = new ErrorResponse(HttpStatus.NOT_ACCEPTABLE, exception.getMessage());
return ResponseEntity.status(HttpStatus.NOT_ACCEPTABLE).body(response);
}
}
The controller code is --
#PostMapping
#ResponseStatus(HttpStatus.CREATED)
#CircuitBreaker(name = "inventory", fallbackMethod = "fallBackMethod")
public String placeOrder(#RequestBody OrderRequest orderRequest) throws OrderItemNotFoundException {
orderService.placeOrder(orderRequest);
return "Order placed successfully";
}
public String fallBackMethod(OrderRequest orderRequest, RuntimeException runtimeException) {
return "The order could not be placed. Please try back after some time.";
}
Edit -- Edited resilience4j.circuitbreaker.instances.inventory.ignore-exceptions[0]=com.mayrevision.orderservice.exception.OrderItemNotFoundException to resilience4j.circuitbreaker.instances.inventory.ignore-exceptions=com.mayrevision.orderservice.exception.OrderItemNotFoundException in application.properties.
Looks like this is how the exception mechanism in Resilience4j is designed to work. If I want the exception to be treated as a success case in all the cases (including the case mentioned above), I should catch it. So I changed the code as follows --
#PostMapping
#CircuitBreaker(name = "inventory", fallbackMethod = "fallBackMethod")
public CompletableFuture<ResponseEntity<StatusResponse>> placeOrder(#RequestBody OrderRequest orderRequest) {
return CompletableFuture.supplyAsync(() -> {
try {
String message = orderService.placeOrder(orderRequest);
StatusResponse response = new StatusResponse(HttpStatus.CREATED, message);
return ResponseEntity.status(HttpStatus.CREATED).body(response);
} catch (OrderItemNotFoundException e) {
return ResponseEntity.status(HttpStatus.NOT_ACCEPTABLE)
.body(new StatusResponse(HttpStatus.NOT_ACCEPTABLE, e.getMessage()));
}
});
}
public CompletableFuture<ResponseEntity<StatusResponse>> fallBackMethod(OrderRequest orderRequest, RuntimeException e) {
return CompletableFuture.supplyAsync(() -> {
//some logic
});
}
This is working close to expected.
I am using hystrix javanica collapser in spring boot, but I found it did not work, my code just like this below:
service class:
public class TestService {
#HystrixCollapser(batchMethod = "getStrList")
public Future<String> getStr(String id) {
System.out.println("single");
return null;
}
#HystrixCommand
public List<String> getStrList(List<String> ids) {
System.out.println("batch,size=" + ids.size());
List<String> strList = Lists.newArrayList();
ids.forEach(id -> strList.add("test"));
return strList;
}
}
where I use:
public static void main(String[] args) {
TestService testService = new TestService();
HystrixRequestContext context = HystrixRequestContext.initializeContext();
Future<String> f1= testService.getStr("111");
Future<String> f2= testService.getStr("222");
try {
Thread.sleep(3000);
System.out.println(f1.get()); // nothing printed
System.out.println(f2.get()); // nothing printed
} catch (Exception e) {
}
context.shutdown();
}
It printed 3 single instead of 1 batch.
I want to know what's wrong with my code, a valid example is better.
I can't find a hystrix javanica sample on the internet, So I have to read the source code to solve this problem, now it's solved, and this is my summary:
when you use hystrix(javanica) collapser in spring-boot, you have to:
Defined a hystrixAspect spring bean and import hystrix-strategy.xml;
Annotate single method with #Hystrix Collapser annotate batch method with #HystrixCommand;
Make your single method need 1 parameter(ArgType) return Future , batch method need List return List and make sure size of args be equal to size of return.
Set hystrix properties batchMethod, scope, if you want to collapse requests from multiple user threads, you must set the scope to GLOBAL;
Before you submit a single request, you must init the hystrix context with HystrixRequestContext.initializeContext(), and shutdown the context when your request finish;
I don't know how to test this and need a quick answer whether it is possible, and how could I possibly test this.
So i have an api-gateway call from a service, and what i want to know is that if following is possible with spring boot
#Component
class GatewayService {
#Autowired
private MicroserviceGateway microServiceGateway;
#Async
#HystrixCommand(threadPool = "microservice", groupKey = "microService", fallback = "getResponse2")
private Future<ResponseDTO> getResponse(RequestDTO request) {
try {
ResponseDTO response = new APIRequest(endpoint, api).body(request).post();
return new AsyncResult(response);
} catch (Exception e) {
throw new GatewayServiceException(e);
}
}
public Future<ResponseDTO> getResponse2(RequestDTO request) {
return Futures.immediateFuture(RequestDTO.empty());
}
}
Will the fallback work? will it all be async?
EDIT : Tried this, and hystrix is being ignored, since the future is returned immediately. Any work arounds? solutions?
I believe the approach to use Hystrix commands asynchronously using a thread pool would be:
If using #HystrixCommand annotation:
...
#HystrixCommand
public Future<Product> findProduct(String id) {
return new AsyncResult<Product>() {
#Override
public Product invoke() {
...
return productService.find ...
}
};
}
...
If using a Hystrix command class, for instance ProductCommandFind.java
...
ProductCommandFind command = new ProductCommandFind(...);
Future<Product> result = command.queue();
...
Hystrix command uses its own and configurable threadpool, maybe configuring a thread pool per command group, lets say a thread pool for command group named PRODUCT_GROUP used by commands: productCommandFind, productCommandCreate, ....
Basically I don't think there is a need to annotate a method with #Async and #HystrixCommand at the same time.
This should work. Your normal DTO method will be executed async and during fallback it will be synchronous.
#Retryable doesn't seem to be working on 2nd level of methods as in sphRemoteCall below. I see that a proxy is created but it is never retried on failures.
Once I moved #Retryable to the 1st level of methods like getSubscriberAccount, it's started working.
Example below:
#Service
public class SphIptvClient extends WebServiceGatewaySupport {
//Works over here
#Retryable(maxAttempts=3, backoff=#Backoff(delay=100))
public GetSubscriberAccountResponse getSubscriberAccount(String loginTocken, String billingServId) {
GetSubscriberAccountResponse response = (GetSubscriberAccountResponse) sphRemoteCall(sphIptvEndPoint, getSubAcc, "xxxxx");
return response;
}
/*
* Retryable is not working on the 2nd level methods in the bean.
* It works only with methods which are called directly from outside
* if there is 2nd level method, like this, Retryable is not working.
*/
//#Retryable
private Object sphRemoteCall(String uri, Object requestPayload, String soapAction) {
log.debug("Calling the sph for uri:{} and soapAction:{}", uri, soapAction);
return getWebServiceTemplate().marshalSendAndReceive(uri, requestPayload, new SoapActionCallback(soapAction));
}
}
#Configuration
#EnableRetry
public class SphClientConfig {
#Bean
public SphIptvClient sphIptvClient() {
SphIptvClient client = new SphIptvClient();
return client;
}
}
So this is a super late answer, but since I've just come here and confronted the same problem (again, after years ago wrestling with transactions) I'll furnish a little more fleshed out solution and hopefully someone will find it useful. Suffice to say that #M. Deinum's diagnosis is correct.
In the above case, and to paraphrase Understanding AOP proxies, any place where SphIptvClient gets autowired will be given a reference to a proxy which Spring Retry will create when #EnableRetry is handled:
"The #EnableRetry annotation creates proxies for #Retryable beans" - Declarative Retry - Spring Retry
Once getSubscriberAccount has been invoked and execution has passed through the proxy and into the #Service instance of the object, no reference to the proxy is known. As a result sphRemoteCall is called as if there were no #Retryable at all.
You could work with the framework by shuffling code around in such a way as to allow getSubscriberAccount to call a proxy-ed sphRemoteCall, which requires a new interface and class implementation.
For example:
public interface SphWebService {
Object sphRemoteCall(String uri, Object requestPayload, String soapAction);
}
#Component
public class SphWebServiceImpl implements SphWebService {
#Retryable
public Object sphRemoteCall(String uri, Object requestPayload, String soapAction) {
log.debug("Calling the sph for uri:{} and soapAction:{}", uri, soapAction);
return getWebServiceTemplate().marshalSendAndReceive(uri, requestPayload, new SoapActionCallback(soapAction));
}
}
#Service
public class SphIptvClient extends WebServiceGatewaySupport {
#Autowired
SphWebService sphWebService;
#Retryable(maxAttempts=3, backoff=#Backoff(delay=100))
public GetSubscriberAccountResponse getSubscriberAccount(String loginTocken, String billingServId) {
GetSubscriberAccountResponse response = (GetSubscriberAccountResponse) this.sphWebService.sphRemoteCall(sphIptvEndPoint, getSubAcc, "xxxxx");
return response;
}
}
#Configuration
#EnableRetry
public class SphClientConfig {
// the #Bean method was unnecessary and may cause confusion.
// #Service was already instantiating SphIptvClient behind the scenes.
}
#Retryable only works on the methods when called directly from other classes.
If you will try to invoke one method with #Retryable annotation from some other method of the same class, it will eventually not work.
// any call from this method to test method will not invoke the retry logic.
public void yetAnotherMethod() {
this.test();
}
// it will work
#Retryable(value = {RuntimeException.class}, backoff = #Backoff(delay = 1500))
public void test() {
System.out.println("Count: " + count++);
throw new RuntimeException("testing");
}
#Recover
public void recover() {
System.out.println("Exception occured.");
}
So, the output if test method is called, will be:
Count: 0
Count: 1
Count: 2
Exception occured.
But, if the yetAnotherMethod is called, output will be:
Count: 0
And a Runtime exception will be thrown.
Suppose you have a method which calls certain API - callAPI() and you want to implement retry logic over it, you can try use a do while, as it will execute only once, if successful.
Method to hit the external API
public int callAPI() {
return 1;
}
Method to implement retry logic
public int retrylogic() throws InterruptedException {
int retry = 0;
int status = -1;
boolean delay = false;
do {
// adding a delay, if you want some delay between successive retries
if (delay) {
Thread.sleep(2000);
}
// Call the actual method, and capture the response,
// and also catch any exception which occurs during the call.
// (Network down/ endpoint not avaliable
try {
status = callAPI();
}
catch (Exception e) {
System.out.println("Error occured");
status = -1;
}
finally {
switch (status) { //now based on error response or any exception you retry again
case HTTPStatus.OK:
System.out.println("OK");
return status;
default:
System.out.println("Unknown response code");
break;
}
retry++;
System.out.println("Failed retry " + retry + "/" + 3);
delay = true;
}
}while (retry < 3);
return status;
}
Here is a simple server application using Bonjour and written in Java. The main part of the code is given here:
public class ServiceAnnouncer implements IServiceAnnouncer, RegisterListener {
private DNSSDRegistration serviceRecord;
private boolean registered;
public boolean isRegistered(){
return registered;
}
public void registerService() {
try {
serviceRecord = DNSSD.register(0,0,null,"_killerapp._tcp", null,null,1234,null,this);
} catch (DNSSDException e) {
// error handling here
}
}
public void unregisterService(){
serviceRecord.stop();
registered = false;
}
public void serviceRegistered(DNSSDRegistration registration, int flags,String serviceName, String regType, String domain){
registered = true;
}
public void operationFailed(DNSSDService registration, int error){
// do error handling here if you want to.
}
}
I understand it in the following way. We can try to register a service calling "registerService" method which, in its turn, calls "DNSSD.register" method. "DNSSD.register" try to register the service and, in general case, it can end up with two results: service was "successfully registered" and "registration failed". In both cases "DNSSD.register" calls a corresponding method (either "serviceRegistered" or "operationFailed") of the object which was given to the DNSSD.register as the last argument. And programmer decides what to put into "serviceRegistered" and "operationFailed". It is clear.
But should I try to register a service from the "operationFailed"? I am afraid that in this way my application will try to register the service too frequently. Should I put some "sleep" or "pause" into "operationFailed"? But in any case, it seems to me, that when the application is unable to register a service it will be also unable to do something else (for example to take care of GUI). Or may be DNSSD.register introduce some kind of parallelism? I mean it starts a new thread but that if I try to register service from "operation Failed", I could generate a huge number of the threads. Can it happen? If it is the case, should it be a problem? And if it is the case, how can I resolve this problem?
Yes, callbacks from the DNSSD APIs can come asynchronously from another thread. This exerpt from the O'Reilly book on ZeroConf networking gives some useful information.
I'm not sure retrying the registration from your operationFailed callback is a good idea. At least without some understanding of why the registration failed, is simply retrying it with the same parameters going to make sense?