We are seeing some errors related to connection reset in Production under heavy load.
Basically the flow is
Microservice 1 -> Load Balancer() -> Microservice 2
Error message is
Stack trace:
|_ checkpoint ? Request to post
http://LoadbalancerURL/process [Default web client]
Error has been observed at the following site(s):
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
My question is how do we troubleshoot this issue.
Do we need to take tcpdump? if yes then what should I look for?
Possible workaround I could think of is use exponential retry. Any idea where I could insert retry logic in below code?
Below is the code snipnet on how we are calling the url from microservice 1
webclient
.create()
.post()
.uri(url)
.bodyValue(input)
.retrieve()
.toEntity(object.class)
.log()
.subscribe(
onSuccess -> {
//somelogic
},
onError -> {
//somelogic
},
()->{
})
Related
I have taken over some code, whee they are using
WebClient.builder().build() to create the WebClient. Rather than WebClient.create() The problem is I am not sure how to use the WebClient created by WebClient.builder().build().
I have the following code
import org.springframework.web.reactive.function.client.WebClient;
Boolean flag = webClient.get()
.uri(url)
.retrieve()
.bodyToMono(Boolean.class)
.block();
This will work if I use WebClient.create()
But using build I get the following exception, so there has to be some functionality I am missing to get WebClient from build working.
org.springframework.web.reactive.function.client.WebClientResponseException$ServiceUnavailable: 503 Service Unavailable from UNKNOWN
at org.springframework.web.reactive.function.client.WebClientResponseException.create(WebClientResponseException.java:207)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ 503 from GET http://localhost:8084/throttling?paramUrl=https://playwright.dev/ [DefaultWebClient]
Stack trace:
at org.springframework.web.reactive.function.client.WebClientResponseException.create(WebClientResponseException.java:207)
at org.springframework.web.reactive.function.client.DefaultClientResponse.lambda$createException$1(DefaultClientResponse.java:206)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:100)
at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:92)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:96)
at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287)
at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:152)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144)
at reactor.core.publisher.FluxJust$WeakScalarSubscription.request(FluxJust.java:101)
I am getting following exeption connecting to Mssql Server.
> org.springframework.dao.DataAccessResourceFailureException: Failed to obtain R2DBC Connection; nested exception is java.net.UnknownHostException: failed to resolve '' after 10 queries
at org.springframework.r2dbc.connection.ConnectionFactoryUtils.lambda$getConnection$0(ConnectionFactoryUtils.java:88) ~[spring-r2dbc-5.3.2.jar:5.3.2]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ Handler com.reactive.testreactive.controller.TestStreamController#findAll() [DispatcherHandler]
|_ checkpoint ⇢ HTTP GET "/test" [ExceptionHandlingWebHandler]
I connected to JDBC with the same configuration in properties but having an issue while trying to connect to R2DBC. Happens on rest and not on starting an app.
#Bean
public MssqlConnectionFactory connectionFactory() {
return new MssqlConnectionFactory(MssqlConnectionConfiguration.builder()
.host("host")
.port(1433)
.database("DataBase")
.username("username")
.password("password")
.build());
}
Do you use absolutely the same configuration?
Because if you are starting using R2DBC you should replace database URL: jdbc:postgresql://... -> r2dbc:postgresql://
If anyone still looking for solution, I'm adding solution below.
First, make sure your database host is visible. By looking at your error statement which says -- failed to resolve '' after 10 queries, means it is empty ''. The way that you have defined the host name is not correct. If you are injecting the host value from properties, you might want to cross check that and then follow the steps below.
If you are configuring through YMLs, you can use below configurations:
spring:
data:
r2dbc:
repositories:
enabled: true
r2dbc:
url: r2dbc:sqlserver://<just_host>:<port>
username: <db_username>
password: <db_password>
name: <db_name>
spring.data.r2dbc.repositories.enabled: true is optional here.
If you are creating custom bean, you can create bean like below. Please note that I'm returning ConnectionFactory rather than MssqlConnectionFactory.
#Bean
public ConnectionFactory connectionFactory() {
return new MssqlConnectionFactory(
MssqlConnectionConfiguration.builder()
.host("just_host")
.database("db_name")
.port(1433)
.username("db_username")
.password("db_password")
.build());
}
I have configured Zuul with Eureka in a way, that 3 identical instances of a service are working parallely. I am calling the gateway on the port 8400, which routes incoming requests to ports 8420, 8430 and 8440 in a round-robin manner. It works smoothly. Now, if I switching off one of the 3 services, a small amount of incoming requests will go wrong with the following exception:
com.netflix.zuul.exception.ZuulException: Filter threw Exception
=> 1: java.util.concurrent.FutureTask.report(FutureTask.java:122)
=> 3: hu.perit.spvitamin.core.batchprocessing.BatchProcessor.process(BatchProcessor.java:106)
caused by: com.netflix.zuul.exception.ZuulException: Filter threw Exception
=> 1: com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:227)
caused by: org.springframework.cloud.netflix.zuul.util.ZuulRuntimeException: com.netflix.zuul.exception.ZuulException: Forwarding error
=> 1: org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:124)
caused by: com.netflix.zuul.exception.ZuulException: Forwarding error
=> 1: org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.handleException(RibbonRoutingFilter.java:198)
caused by: com.netflix.client.ClientException: com.netflix.client.ClientException
=> 1: com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:118)
caused by: java.lang.RuntimeException: org.apache.http.NoHttpResponseException: scalable-service-2:8430 failed to respond
=> 1: rx.exceptions.Exceptions.propagate(Exceptions.java:57)
caused by: org.apache.http.NoHttpResponseException: scalable-service-2:8430 failed to respond
=> 1: org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
My Zuul routing looks like this:
### Zuul routes
zuul.routes.scalable-service.path=/scalable/**
#Authorization header will be forwarded to scalable-service
zuul.routes.scalable-service.sensitiveHeaders: Cookie,Set-Cookie
zuul.routes.scalable-service.serviceId=template-scalable-service
It takes a while until Eureka discovers the service is not available any more.
My question is: Is there a possibility, to configure Zuul so that in case of a NoHttpResponseException, it forwards the requests to another available instance in the pool?
Eureka, by default, requires lease to be renewed every 90s. That is, if a service instance doesn't get its lease renewed in 90s, Eureka server will evict the instance. In your case, the instance has not been evicted yet - the renew window for the instance was valid.
For this, you can decrease the renew duration through config setup at eureka client and eureka server as described here.
Note: If you hit the actuator /shutdown endpoint, the instance is immediately evicted
Finally I found the solution to the problem. The appropriate search phrase was 'fault tolerance'. The key is the autoretry config in the following application.properties file. The value of template-scalable-service.ribbon.MaxAutoRetriesNextServer must be set at least to 6 in case of 3 pooled services to achieve full fault tolerance. With that setup I can kill 2 of 3 services any time, no incoming request will go wrong. Finally I have set it to 10, there is no unnecessary increase of timeout, hystrix will break the line.
### Eureka config
eureka.instance.hostname=${hostname:localhost}
eureka.instance.instanceId=${eureka.instance.hostname}:${spring.application.name}:${server.port}
eureka.instance.non-secure-port-enabled=false
eureka.instance.secure-port-enabled=true
eureka.instance.secure-port=${server.port}
eureka.instance.lease-renewal-interval-in-seconds=5
eureka.instance.lease-expiration-duration-in-seconds=10
eureka.datacenter=perit.hu
eureka.environment=${EUREKA_ENVIRONMENT_PROFILE:dev}
eureka.client.serviceUrl.defaultZone=${EUREKA_SERVER:https://${server.fqdn}:${server.port}/eureka}
eureka.client.server.waitTimeInMsWhenSyncEmpty=0
eureka.client.registry-fetch-interval-seconds=5
eureka.dashboard.path=/gui
eureka.server.enable-self-preservation=false
eureka.server.expected-client-renewal-interval-seconds=10
eureka.server.eviction-interval-timer-in-ms=2000
### Ribbon
ribbon.IsSecure=true
ribbon.NFLoadBalancerPingInterval=5
ribbon.ConnectTimeout=30000
ribbon.ReadTimeout=120000
### Zuul config
zuul.host.connectTimeoutMillis=30000
zuul.host.socketTimeoutMillis=120000
zuul.host.maxTotalConnections=2000
zuul.host.maxPerRouteConnections=200
zuul.retryable=true
### Zuul routes
#template-scalable-service
zuul.routes.scalable-service.path=/scalable/**
#Authorization header will be forwarded to scalable-service
zuul.routes.scalable-service.sensitiveHeaders=Cookie,Set-Cookie
zuul.routes.scalable-service.serviceId=template-scalable-service
# Autoretry config for template-scalable-service
template-scalable-service.ribbon.MaxAutoRetries=0
template-scalable-service.ribbon.MaxAutoRetriesNextServer=10
template-scalable-service.ribbon.OkToRetryOnAllOperations=true
#template-auth-service
zuul.routes.auth-service.path=/auth/**
#Authorization header will be forwarded to scalable-service
zuul.routes.auth-service.sensitiveHeaders=Cookie,Set-Cookie
zuul.routes.auth-service.serviceId=template-auth-service
# Autoretry config for template-auth-service
template-auth-service.ribbon.MaxAutoRetries=0
template-auth-service.ribbon.MaxAutoRetriesNextServer=0
template-auth-service.ribbon.OkToRetryOnAllOperations=false
### Hystrix
hystrix.command.default.execution.timeout.enabled=false
Beside of this, I have a profile specific setup in application-discovery.properties
#Microservice environment
eureka.client.registerWithEureka=false
eureka.client.fetchRegistry=true
spring.cloud.loadbalancer.ribbon.enabled=true
I start my server in a docker container like this:
services:
discovery:
container_name: discovery
image: template-eureka
environment:
#agentlib for remote debugging
- JAVA_OPTS=-DEUREKA_SERVER=https://discovery:8400/eureka -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
- TEMPLATE_EUREKA_OPTS=-Dspring.profiles.active=default,dev,discovery
- EUREKA_ENVIRONMENT_PROFILE=dev
ports:
- '8400:8400'
- '5500:5005'
networks:
- back-tier-net
- monitoring
hostname: 'discovery'
See the complete solution in GitHub.
Stack:AngularJS v1.6.5, java 8, spring boot, tomcat.
After about 1 week of work , the application not response with such an error. Why this happening?
Frontend:
$http({
url: 'find',
method: "post",
data: { 'month' : $scope.month,'year' : $scope.year, 'payTime' : $scope.payTime,'waitTime' : $scope.waitTime,'scanTime' : $scope.scanTime,'gbNumber' : $scope.hyper}
})
.then(function(response) {
..
});
}
Backend:
#RequestMapping(path = "/find", method = RequestMethod.POST)
public ReportResponse find(#RequestBody RequestSearch params,
HttpServletResponse response) throws DataNotFoundException {
...
}
Stacktrace:
2018-04-02 09:37:44.738 ERROR 14912 --- [p-nio-80-exec-9] o.s.boot.web.support.ErrorPageFilter : Cannot forward to error page for request [/excel/ExceReport.xls] as the response has already been committed. As a result, the response may have the wrong status code. If your application is running on WebSphere Application Server you may be able to resolve this problem by setting com.ibm.ws.webcontainer.invokeFlushAfterService to false
org.apache.catalina.connector.ClientAbortException: java.io.IOException: An established connection was aborted by the software in your host machine
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:356) ~[catalina.jar:8.5.24]
at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:815) ~[catalina.jar:8.5.24]
at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:720) ~[catalina.jar:8.5.24]
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:391) ~[catalina.jar:8.5.24]
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:369) ~[catalina.jar:8.5.24]
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96) ~[catalina.jar:8.5.24]
at org.springframework.util.StreamUtils.copy(StreamUtils.java:138) ~[spring-core-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.http.converter.ResourceHttpMessageConverter.writeContent(ResourceHttpMessageConverter.java:110) ~[spring-web-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.http.converter.ResourceHttpMessageConverter.writeInternal(ResourceHttpMessageConverter.java:102) ~[spring-web-4.3.9.RELEASE.jar:4.3.9.RELEASE]
...
Cause
This exception can mean that the connection to the client browser was
aborted before the response is fully transferred. It is a harmless
warning as it can be due to transient network problems or the user
aborts/refreshes the page before it loaded.
A list of other causes are:
The user closed the browser before the page loaded.
Their Internet connection failed during loading.
They went to another page before the page loaded.
The browser timed the connection out before the page loaded (would
have to be a large page).
Resolution
This can be ignored, unless there are other issues that are currently
occurring. For example, if the your application server is throwing a
lot of these, it might be a sign of a performance problem.
I have an app that exposes Websocket/SockJS/Stomp server endpoints and would like to run a JUnit tests that runs client (Java STOMP client, also from Spring) against it, to test "sending" features.
I have a test like
public void measureSpeedWithWebsocket() throws Exception {
final Waiter subscribeWaiter = new Waiter();
new Thread(() -> {
// Prepare connection
WebsocketClient listener = new WebsocketClient("/mytopic/stomp");
try {
listener.connectAndSubscribe();
subscribeWaiter.resume();
} catch (Exception e) {
e.printStackTrace();
}
}).start();
subscribeWaiter.await(); // Wait for connection.
Here I made use of Waiter from https://github.com/jhalterman/concurrentunit, which effect is basically to delay main thread of the test till secondary thread call resume(). This is likely wrong, because Spring server that is running in the context has to react
I am getting the following error
[INFO ] 2017-02-03 12:36:12.402 [Thread-19] WebsocketClient - Listening
[INFO ] 2017-02-03 12:36:12.403 [Thread-19] WebsocketClient - Connecting to ws://localhost:8083/user...
2017-02-03 12:36:14.097 ERROR 9956 --- [ Thread-19] o.s.w.socket.sockjs.client.SockJsClient : Initial SockJS "Info" request to server failed, url=ws://localhost:8083/user
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8083/user/info": Connection refused: connect; nested exception is java.net.ConnectException: Connection refused: connect
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:633) ~[spring-web-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:595) ~[spring-web-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.web.socket.sockjs.client.RestTemplateXhrTransport.executeInfoRequestInternal(RestTemplateXhrTransport.java:138) ~[spring-websocket-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.web.socket.sockjs.client.AbstractXhrTransport.executeInfoRequest(AbstractXhrTransport.java:155) ~[spring-websocket-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.web.socket.sockjs.client.SockJsClient.getServerInfo(SockJsClient.java:286) ~[spring-websocket-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.web.socket.sockjs.client.SockJsClient.doHandshake(SockJsClient.java:254) ~[spring-websocket-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.web.socket.messaging.WebSocketStompClient.connect(WebSocketStompClient.java:274) [spring-websocket-4.3.3.RELEASE.jar:4.3.3.RELEASE]
at org.springframework.web.socket.messaging.WebSocketStompClient.connect(WebSocketStompClient.java:255) [spring-websocket-4.3.3.RELEASE.jar:4.3.3.RELEASE]
(...)
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_74]
Caused by: java.net.ConnectException: Connection refused: connect
How I can possibly make a proper test that "self-connects" to the websocket offered by my Spring Boot application?
If you are using Spring Boot than you are a lucky one :).
Here is an example http://rafaelhz.github.io/testing-websockets/ how you can test the web sockets, which perfectly works in Spring Boot and can helps you a lot. I am trying to do the same in Spring MVC but unfortunately that doesn't work in Spring MVC.