I am currently using Spring Cloud and Feign to consume a Microservice in my application. Since it can happen, that a database connection or the like fails in a single service instance, making it return 500 HTTP status code, I want to make sure, that the next server is retried by the service's clients. Currently, Ribbon's retry mechanism works like a charm when the service is not running at all, however it still returns instantly an error when it receives a 500 status code, without any retry.
Is it possible to configure the Feign clients or their underlying Ribbon load balancers to retry the next server, if an instance returns a 500 response?
The configuration is pretty much the same as in this thread: Does Feign retry require some sort of configuration?
I would love to use an implementation like Ribbons' HttpResponseValidator (https://github.com/Netflix/ribbon/blob/master/ribbon/src/main/java/com/netflix/ribbon/http/HttpResponseValidator.java), but I couldn't find anything usable for Spring Cloud and its Feign/Ribbon integration
This question is very old and the solution was probably already found or wasn't possible at the time. Anyway, I think that answer might still help someone 8 ).
Please use this as a reference, this code is not intended for production use.
Feign allows you to configure errorDecoder - this is the place where magic happens.
Feign.Builder builder = Feign.builder()
.errorDecoder(new RetryOnScaleErrorDecoder())
Here is the implementation, I use that class to retry request on HTTP error 429 I get from AWS when service is scaling
public static class RetryOnScaleErrorDecoder implements ErrorDecoder {
#Override
public Exception decode(String methodKey, Response response) {
FeignException exception = errorStatus(methodKey, response);
// This is a terrible part please check how feign.codec.ErrorDecoder.RetryAfterDecoder is implemented for proper parsing of retry-after header
Collection<String> headers = response.headers().get("Retry-After");
String repeatAfterString = "0";
if (Objects.nonNull(headers)) {
repeatAfterString = Iterables.getFirst(headers, "0");
}
assert repeatAfterString != null;
Date repeatAfter = new Date(currentTimeMillis());
if (repeatAfterString.matches("^[0-9]+$")) {
try {
long deltaMillis = SECONDS.toMillis(Long.parseLong(repeatAfterString));
repeatAfter = new Date(currentTimeMillis() + deltaMillis);
} catch (NumberFormatException ignored) {
// TODO: logging
}
}
// That's the part where we decide to retry based on status code value
if (exception.status() == 429) {
return new RetryableException(
response.status(),
exception.getMessage(),
response.request().httpMethod(),
exception,
repeatAfter
);
}
return exception;
}
}
I think that in conjunction with Ribbon it will produce desired result.
Try to this config:
MY-SPRING-API.ribbon.retryableStatusCodes=404,500
This is the same question:
Feign client and Spring retry
document is :
https://docs.spring.io/spring-cloud-netflix/docs/2.2.10.RELEASE/reference/html/#retrying-failed-requests
Related
In the latest versions of spring cloud Netflix's official documentation (for example 2.0.2.RELEASE, the last GA version) it states:
When Spring Retry is present, load-balanced RestTemplates, Feign, and Zuul automatically retry any failed requests (assuming your configuration allows doing so).
But that seems wrong concerning the use of standalone Ribbon (i.e. load-balanced RestTemplates). I couldn't make it work nor found any working example. Moreover, I found other sources stating quite the opposite was true, such as:
spring-cloud with RestTemplate//Ribbon/Eureka - retry when server not available
Spring Cloud - Getting Retry Working In RestTemplate?).
So, it's the documentation wrong or is the whole world missing something?
We found the same issue; Zuul will neither retry nor failover to alternative servers in the ribbon server list using the default (Apache Http Client) configuration. We tracked it down to this line of code in the RibbonLoadBalancingHttpClient class:
#Override
public RequestSpecificRetryHandler getRequestSpecificRetryHandler(
RibbonApacheHttpRequest request, IClientConfig requestConfig) {
return new RequestSpecificRetryHandler(false, false, RetryHandler.DEFAULT,
requestConfig);
}
The hardcoded false, false parameters effectively disable retry from ever happening. It is easy to fix this.
Add this to your Zuul main application class:
#RibbonClients(
defaultConfiguration = {EurekaRibbonClientConfiguration.class,
MyRibbonConfiguration.class})
Create a class, MyRibbonConfiguration and use it to emulate the ribbonLoadBalancingHttpClient method here except we override and fix the getRequestSpecificRetryHandler method that's causing problems.
public class MyRibbonConfiguration {
#RibbonClientName
private String name = "client";
#Bean
public RibbonLoadBalancingHttpClient ribbonLoadBalancingHttpClient(
IClientConfig config, ServerIntrospector serverIntrospector,
ILoadBalancer loadBalancer, RetryHandler retryHandler, CloseableHttpClient httpClient) {
RibbonLoadBalancingHttpClient client =
new RibbonLoadBalancingHttpClient(httpClient, config, serverIntrospector) {
#Override
public RequestSpecificRetryHandler getRequestSpecificRetryHandler(
RibbonApacheHttpRequest request, IClientConfig requestConfig) {
return new RequestSpecificRetryHandler(true, true, RetryHandler.DEFAULT,
requestConfig);
}
};
client.setLoadBalancer(loadBalancer);
client.setRetryHandler(retryHandler);
Monitors.registerObject("Client_" + this.name, client);
return client;
}
}
Adjust true, true to reference from properties if you want. With this in place Zuul will start respecting the ribbon.MaxAutoRetries and ribbon.MaxAutoRetriesNextServer properties.
Note that if you are using Ribbon with Feign for server->server calls then those Feign calls are not affected because OpenFeign hardcodes those two parameters to true, true.
I am using jetty with spring boot, and implemented few rest calls, all the get methods are working fine in both the environment dev and qa , but post is always returning '405 Method Not Allowed', I checked everything like headers, body and other parameters everything seems correct.
I am using spring rest implementation, any ideas why its failing.
method Definition:
#RequestMapping(value ="/resub" ,method=RequestMethod.POST )
public ResponseEntity<?> resub(#RequestBody Subscription Subscription,HttpServletRequest request ){
try {
// call to other service methods
//}
} catch (Exception e) {
if(e instanceof HttpClientErrorException)
return commonUtil.getExceptionResponse(e.getMessage());
else
return commonUtil.getExceptionResponse("Unknown Error Occured"); }
}
This answer should really be a comment, but unfortunately I don't have 50 rep yet.
The first problem I notice is that you're closing the resub() function before it even gets to the catch statement.
That doesn't explain why you would get a 405 http status response though.
"The principal cause of this issue is either using the wrong method (not applicable to your case) or multiple handlers have been defined for the same verb/method, and one of the handlers is blocking the expected handler from processing the request. ... IIS processes handlers from first to last based on the order handler entries in the applicationHost.config and web.config files, where the first matching combination of path, verb, resource, etc., will be used to handle the request [1]."
Hope this helps, if not let me know and I can continue trying to help :)
[1] - https://www.asp.net/web-api/overview/testing-and-debugging/troubleshooting-http-405-errors-after-publishing-web-api-applications
Hey i have an http outbound gateway which i've attached a retry advice to:
.handle(Http.outboundGateway(spelParser.parseExpression("headers." + HeaderKeys.TARGET_ENDPOINT))
.extractPayload(true)
.httpMethod(HttpMethod.POST)
.expectedResponseType(String.class)
.requestFactory(requestFactory())
.get()
, httpOutboundEndpointSpec())
The httpOutboundEndpointSpec:
#Bean
public Consumer<GenericEndpointSpec<HttpRequestExecutingMessageHandler>> httpOutboundEndpointSpec() {
return new Consumer<GenericEndpointSpec<HttpRequestExecutingMessageHandler>>() {
#Override
public void accept(GenericEndpointSpec<HttpRequestExecutingMessageHandler> spec) {
spec.advice(context.getBean("lengthy", RequestHandlerRetryAdvice.class));
spec.requiresReply(true);
}
};
}
Is there a way to hook up the advice dynamically depending on the contents of the message? Different clients now need different intervals of backoff. I could write one outbound gateway per client with a different retry advice each, but that would make a lot of gateways.
Probably the simplest solution would be a custom advice that has a set of RetryTemplates and uses one of them based on the message.
You could base your advice on the and choose which retry template to use.
When I have my application on tomcat or weblogic and having the following controller:
#ResponseBody
#RequestMapping("/loadByReceiptNo/{receiptNo}/{cashId}")
public Securities loadByReceiptNo(#PathVariable int receiptNo, #PathVariable int cashId) {
return myService.loadByReceiptNo(receiptNo, cashId);
}
calling
/loadByReceiptNo/1
or
/loadByReceiptNo/1/
cause the server hanging for too long and it needs a restart.
how should I avoid such a situation?
UPDATE
this url is being called by another machine and this url serves as webservice. Getting dump threat, I realized that this part of client stuck in deadlock:
conn.getResponseCode()
The temporal relief to this issue is using:
conn.setConnectTimeout(10000);
conn.setReadTimeout(10000);
Any other solutions will be appreciated.
You are providing wrong URL - /loadByReceiptNo/1. You should also provide {cashId} and not just {receiptNo}. So your URL should be something like /loadByReceiptNo/1/1.
Reason being #Pathvariable cannot be null. You would probably want to do something like -
#ResponseBody
#RequestMapping(value={"/loadByReceiptNo/{receiptNo}/{cashId}","/loadByReceiptNo/{receiptNo}"})
public Securities loadByReceiptNo(#PathVariable Map<String, String> pathVariablesMap) {
if (pathVariables.containsKey("receiptNo") && pathVariables.containsKey("cashId")) {
myService.loadByReceiptNo(receiptNo, cashId);
} else {
//probably return error response
}
}
or write a new controller method altogether for just /loadByReceiptNo/{receiptNo}.
OR you can make cashId as #RequestParam(value="cashId ",required=false) and make it optional.
Either ways your current Mapping is incorrect. However these thing should not let you server hang. So either there is a deadlock or your service/dao call is taking too long. If that is the case I suggest use asynchronous processing like using activemq and not hold server threads. Just return after validation with error or success code and then process the request from the queue.
I have a Spring Boot REST service that sometimes call third party services as a part of a request. I would like to set a timeout on all my resources (let's say 5 seconds), so that if any request handling (the whole chain, from incoming to response) takes longer than 5 seconds my controllers responds with HTTP 503 instead of the actual response. It would be awesome if this was just a Spring property, for example setting
spring.mvc.async.request-timeout=5000
but I haven't had any luck with that. I've also tried extending WebMvcConfigurationSupport and overriding configureAsyncSupport:
#Override
public void configureAsyncSupport(final AsyncSupportConfigurer configurer) {
configurer.setDefaultTimeout(5000);
configurer.registerCallableInterceptors(timeoutInterceptor());
}
#Bean
public TimeoutCallableProcessingInterceptor timeoutInterceptor() {
return new TimeoutCallableProcessingInterceptor();
}
without any luck.
I suspect I have to manually time all my third party calls, and if they take too long, throw a timeout exception. Is that right? Or is there any easier, holistic solution that covers all my request endpoints?
You need to return a Callable<> if you want spring.mvc.async.request-timeout=5000 to work.
#RequestMapping(method = RequestMethod.GET)
public Callable<String> getFoobar() throws InterruptedException {
return new Callable<String>() {
#Override
public String call() throws Exception {
Thread.sleep(8000); //this will cause a timeout
return "foobar";
}
};
}
The #Transactional annotation takes a timeout parameter where you can specify timeout in seconds for a specific method in the #RestController
#RequestMapping(value = "/method",
method = RequestMethod.POST,
produces = MediaType.APPLICATION_JSON_VALUE)
#Timed
#Transactional(timeout = 120)
A fresh answer for Spring Boot 2.2 is required as server.connection-timeout=5000 is deprecated. Each server behaves differently, so server specific properties are recommended instead.
SpringBoot embeds Tomcat by default, if you haven't reconfigured it with Jetty or something else. Use server specific application properties like server.tomcat.connection-timeout or server.jetty.idle-timeout.
As commented by Wilkinson:
Setting the connection timeout will only result in a timeout when the
client connects but is then too slow to send its request. If you want
the client to wait for a maximum of 30 seconds for a response you will
have to configure that on the client-side. If you want the server-side
to only spend a maximum of 30 seconds handling the request there is no
way to guarantee that as you cannot force the thread that is handling
the request to stop.
You can also try setting spring.mvc.async.request-timeout
I would suggest you have a look at the Spring Cloud Netflix Hystrix starter to handle potentially unreliable/slow remote calls. It implements the Circuit Breaker pattern, that is intended for precisely this sorta thing.
See offcial docs for more information.
You can try server.connection-timeout=5000 in your application.properties. From the official documentation:
server.connection-timeout= # Time in milliseconds that connectors will
wait for another HTTP request before closing the connection. When not
set, the connector's container-specific default will be used. Use a
value of -1 to indicate no (i.e. infinite) timeout.
On the other hand, you may want to handle timeouts on the client side using Circuit Breaker pattern as I have already described in my answer here: https://stackoverflow.com/a/44484579/2328781
if you are using RestTemplate than you should use following code to implement timeouts
#Bean
public RestTemplate restTemplate() {
return new RestTemplate(clientHttpRequestFactory());
}
private ClientHttpRequestFactory clientHttpRequestFactory() {
HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory();
factory.setReadTimeout(2000);
factory.setConnectTimeout(2000);
return factory;
}}
The xml configuration
<bean class="org.springframework.web.client.RestTemplate">
<constructor-arg>
<bean class="org.springframework.http.client.HttpComponentsClientHttpRequestFactory"
p:readTimeout="2000"
p:connectTimeout="2000" />
</constructor-arg>
In Spring properties files, you can't just specify a number for this property. You also need to specify a unit. So you can say spring.mvc.async.request-timeout=5000ms or spring.mvc.async.request-timeout=5s, both of which will give you a 5-second timeout.
I feel like none of the answers really solve the issue. I think you need to tell the embedded server of Spring Boot what should be the maximum time to process a request. How exactly we do that is dependent on the type of the embedded server used.
In case of Undertow, one can do this:
#Component
class WebServerCustomizer : WebServerFactoryCustomizer<UndertowServletWebServerFactory> {
override fun customize(factory: UndertowServletWebServerFactory) {
factory.addBuilderCustomizers(UndertowBuilderCustomizer {
it.setSocketOption(Options.READ_TIMEOUT, 5000)
it.setSocketOption(Options.WRITE_TIMEOUT, 25000)
})
}
}
Spring Boot official doc: https://docs.spring.io/spring-boot/docs/2.2.0.RELEASE/reference/html/howto.html#howto-configure-webserver
You can configure the Async thread executor for your Springboot REST services. The setKeepAliveSeconds() should consider the execution time for the requests chain. Set the ThreadPoolExecutor's keep-alive seconds. Default is 60. This setting can be modified at runtime, for example through JMX.
#Bean(name="asyncExec")
public Executor asyncExecutor()
{
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(3);
executor.setMaxPoolSize(3);
executor.setQueueCapacity(10);
executor.setThreadNamePrefix("AsynchThread-");
executor.setAllowCoreThreadTimeOut(true);
executor.setKeepAliveSeconds(10);
executor.initialize();
return executor;
}
Then you can define your REST endpoint as follows
#Async("asyncExec")
#PostMapping("/delayedService")
public CompletableFuture<String> doDelay()
{
String response = service.callDelayedService();
return CompletableFuture.completedFuture(response);
}