Am facing the below ZuulException exception due to SocketTimeoutException Read timed out. I am trying to put my oauth2 server behind the zuul proxy.
Please see the log trace here , gateway's application.yml entries here and application dependencies here . I am not using hystrix or eureka explicitly
This issue is intermittent, sometimes it is working and sometimes it isn't. Have anyone faced this before.
everything works well except API gateway.
Try to define the below properties. It seems that you're using Zuul with Eureka. In this case, RibbonRoutingFilter will be used instead of SimpleHostRoutingFilter. If so, you need to define readTimeout and connectTimeout for ribbon instead of zuul.host properties.
ribbon:
ReadTimeout: 10000
ConnectTimeout: 10000
oauth2:
ribbon:
ReadTimeout: 10000
ConnectTimeout: 10000
Related
I have the following setup:
Spring Cloud Gateway on Port 8080
Routing /user to Spring Rest API (Called User) on Port 9000
Routing /character to Spring Rest API (Called Character) on Port 9001
Spring Cloud Config Server on Port 8090
The Gateway and the two Rest API applications are connected to the Cloud Config to pull the configs. The applications themselves start up correctly and can be used via their respective ports. I tested this by calling /swagger-ui.html which works fine.
When calling /actuator/gateway/routes I get a list of the routes of the gateway, which seem fine.
For all four applications I set up:
server:
address: 0.0.0.0
port: 8080 # ports are adjusted for each service
forward-headers-strategy: framework
When I did not use the Spring Cloud Config Server the Gateway was running fine. Now I'm receiving the following error whenever calling either /user/swagger-ui.html or /character/swagger-ui.html
java.lang.IllegalArgumentException: Invalid IPv4 address: 0:0:0:0:0:0:0:1:60979
at org.springframework.web.util.UriComponentsBuilder.parseForwardedFor(UriComponentsBuilder.java:363) ~[spring-web-5.3.8.jar:5.3.8]
at org.springframework.web.filter.ForwardedHeaderFilter$ForwardedHeaderExtractingRequest.<init>(ForwardedHeaderFilter.java:246) ~[spring-web-5.3.8.jar:5.3.8]
at org.springframework.web.filter.ForwardedHeaderFilter.doFilterInternal(ForwardedHeaderFilter.java:149) ~[spring-web-5.3.8.jar:5.3.8]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar:5.3.8]
Tried changing forward-headers-strategy to native which did not help either. I dug into the UriComponentBuilder and set some breakpoints to investigate the variables in their and I suspect request.headers["forwarded"] is the reason it fails:
It tries to resolve the "for" inside the variable which is not a IPv4 address and thus fails. Forcing ipv4 with "server.address" seemed to work before using Spring Cloud Config, does not now, however. Has anyone come across the same issue and knows what am I supposed to do to get rid off this exception?
So all in all the routing itself seems to work, as it routes to the correct application. The application itself (User in this case) throws the error, not the Gateway.
The issue is resolved with
Spring gateway version 3.0.4
which you can get from
org.springframework.cloud:spring-cloud-dependencies:2020.0.4
Unable to deregister the service from Consul.
Basically Consul official page said that it will deregister service automatically but in my case it won't work like said.
https://www.consul.io/docs/agent/basics.html
Hi Referring to the consul Life-cycle its says that
To prevent an accumulation of dead nodes (nodes in either failed or left states), Consul will automatically remove dead nodes out of the catalog. This process is called reaping. This is currently done on a configurable interval of 72 hours (changing the reap interval is not recommended due to its consequences during outage situations). Reaping is similar to leaving, causing all associated services to be deregistered.
This is my bootstrap.yml file
server:
port: 8089
spring:
application:
name: ***-service
cloud:
consul:
host: consul-ui
port: 8500
discovery:
deregister: true
instance-id: ${spring.application.name}:${random.value}
enabled: true
register: true
health-check-interval: 20s
prefer-ip-address: true
config:
enabled: true
prefix: configuration
defaultContext: shared
format: YAML
data-key: data
watch:
enabled: true
endpoints:
shutdown:
enabled: true
In Consul UI after deleting service using purge command, still shows on Consul UI. So meant that it is not been deregister from Consul
You need to configure this on Consul, as your apps seems to not exit gracefully.
Checkout the consul config property deregister_critical_service_after:
In Consul 0.7 and later, checks that are associated with a service may
also contain an optional deregister_critical_service_after field,
which is a timeout in the same Go time format as interval and ttl. If
a check is in the critical state for more than this configured value,
then its associated service (and all of its associated checks) will
automatically be deregistered. The minimum timeout is 1 minute, and
the process that reaps critical services runs every 30 seconds, so it
may take slightly longer than the configured timeout to trigger the
deregistration. This should generally be configured with a timeout
that's much, much longer than any expected recoverable outage for the
given service.
That documentation is about consul nodes, not service nodes.
How exactly are you terminating the application?
I am attempting to setup a Spring Cloud Config Server backed by Spring Cloud Vault secret management. I'm relatively new to Spring but I have tried following instructions and examples here:-
http://cloud.spring.io/spring-cloud-vault-config/
Everything works fine provided you follow the default settings like http, localhost and 8200 for the vault endpoint and tls_disable = 1 to switch off SSL. However these are not practical settings for any real environment and there are few examples anywhere that help with this. Can anyone help with a working example?
I have Successfully set up vault with TLS enable. I have successfully set up a config server that connects using a self signed cert. I can even inject a secret value into the config server and expose it via #Value and #PostConstruct.
All of this is working. However when I try to leverage Spring Conig endpoints to access vault, I get the following:-
{
"timestamp": 1486413850574,
"status": 500,
"error": "Internal Server Error",
"exception": "org.springframework.web.client.ResourceAccessException",
"message": "I/O error on GET request for \"http://127.0.0.1:8200/v1/secret/myapp\": Connection refused: connect; nested exception is java.net.ConnectException: Connection refused: connect",
"path": "/myapp/default"
}
Config server is using default values even though I have set overrides in the bootstrap.yml.:-
server:
port: 8888
spring:
profiles:
active: vault
spring.cloud.vault:
host: myhost.mydomain.com
port: 8200
scheme: https
authentication: TOKEN
token: 0f1887c3-d8a8-befd-a5a2-01e4e066c50
ssl:
trust-store: configTrustStore.jks
trust-store-password: changeit
As you can see it should be pointing to myhost.mydomain.com not
127.0.0.1 and it should be using https, not http as the protocol scheme.
I'm not sure why it uses these defaults for config server endpoints but uses correct settings during spring cloud vault startup. I'm using all the latest stable builds of Spring Dalsten.M1 and Spring Cloud Vault 1.0.0.M1. I realize these are milestone releases. I've also tried Camden and Brixton combos with no luck. I can supply code if required.
Any help greatly appreciated.
As I mention in my response to spensergibb, I have had some success in resolving this myself. Based on his comments I will clarify my intent as it will help with a common understanding of the issue. I am attempting to do two things:-
Stand up a configuration server that uses Vault as a backend, (as opposed to the default GIT backend) and expose the Vault API to client applications (over TLS) so that they can retrieve their own secrets. I do not want all my client applications to connect to Vault directly. I want them to get their configuration from a config server by having the config server connect to Vault. Until last night I was unable to achieve this goal, unless I set everything up as default with TLS disabled and using loopback address, port 8200 for the Vault software etc. Obviously defaults are not practical for any of our deployed environments. I will mention that the link posted by spencergibb does help me understand why this was not working but the subtlety of the reason is why I missed it before. Read on for my explanation.
I want the config server to configure itself from Vault directly. That is, connect to Vault via Spring Cloud Vault Config. This worked right away for me as described in the documentation. However this goal is somewhat trivial as I do not have a real use case at this time. But I wanted to understand if it could be done since I saw no real reason why not and it seemed like good first steps in integrating Vault.
The distinction between these two capabilities helped me understand that the problem derives from the fact that Spring Cloud Config Server and Spring Cloud Vault appear to be using two different beans to inject the Vault configuration properties. Spring Cloud Config Server uses VaultEnvironmentRepository annotated with #ConfigurationProperties("spring.cloud.config.server.vault") and Spring Cloud Vault uses VaultProperties annotated with #ConfigurationProperties("spring.cloud.vault").
This caused me to add two different configs to my bootstrap yml.
server:
port: 8888
spring:
profiles:
active: local, vault
application:
name: quoting-domain-configuration-server
cloud:
vault:
host: VDDP03P-49A26EF.lm.lmig.com
port: 8200
scheme: https
authentication: TOKEN
token: 0f1997c3-d8a8-befd-a5a2-01e4e066c50a
ssl:
trust-store: configTrustStore.jks
trust-store-password: changeit
config:
server:
vault:
host: VDDP03P-49A26EF.lm.lmig.com
port: 8200
scheme: https
authentication: TOKEN
token: 0f1997c3-d8a8-befd-a5a2-01e4e066c50a
Note the same config details. Just different yml paths. This is the subtle point I missed given that I started by getting goal number 1 to work first and assuming the same config would work for both goals. (Note: Token and password are contrived).
This almost worked except for an SSL handshake error. As you can see there are no SSL attributes set on the spring.cloud.config.server.vault path. The VaultProperties bean does not support them. I was not sure how to deal with this (perhaps another non-vault specific bean that I could not find). My solution was to simply force the cert configuration myself like this:-
#SpringBootApplication
#EnableConfigServer
public class Application
{
public static void main(String[] args)
{
System.setProperty("javax.net.ssl.trustStore",
Application.class.getResource("/configTrustStore.jks").getFile());
System.setProperty("javax.net.ssl.trustStorePassword", "changeit");
SpringApplication.run(Application.class, args);
}
}
This SSL solution is pretty ugly. I'm sure there must be a better way to do this part. So I am open to other suggestions. However once I completed all above steps everything now works.
Thanks for your write up. I was struggling to get this working. I was able to get a client service to connect to a Spring Cloud Config server and a Vault server but I was not able to get a Spring Cloud Config server to connect to a Vault server.
I even struggled after copying your configuration into my Spring Cloud Config server. While I eventually got it working with your configuration I was able to pare it down quite a bit. The key was that the token does not belong in the Spring Cloud Config server. It belongs in the client.
I was trying http://localhost:8888/{application}/default in the browser but got the following:
Whitelabel Error Page
This application has no explicit mapping for /error, so you are seeing this as a fallback.
Thu May 11 14:21:31 EDT 2017
There was an unexpected error (type=Bad Request, status=400).
Missing required header: X-Config-Token
I used PostMan to send the request with a X-Config-Token header containing the Vault token and it worked.
Here is my final config.
server:
port: ${PORT:8888}
management:
context-path: /manage
security:
enabled: true
spring:
profiles:
active: git,vault
application:
name: config-server
cloud:
config:
server:
git:
order: 1
uri: file:///temp/config-server/config
vault:
order: 0
host: localhost
port: 8200
scheme: http
So it looks like you need to add the token to the client. Maybe using spring.cloud.config.token.
instead
#SpringBootApplication
#EnableConfigServer
public class Application
{
public static void main(String[] args)
{
System.setProperty("javax.net.ssl.trustStore",
Application.class.getResource("/configTrustStore.jks").getFile());
System.setProperty("javax.net.ssl.trustStorePassword", "changeit");
SpringApplication.run(Application.class, args);
}
}
write
bootstrap yml ->
javax.net.ssl.trustStore: /configTrustStore.jks
javax.net.ssl.trustStorePassword: changeit
Although I am answering late. But I was able to configure spring cloud config server to use Vault as backend with CERT authentication via certificates. And moreover you do not require to send X-Config-Token in GET request. So your config-server GET requests will work in the same way it works with GITHUB as backend.As my implementation will get the token on the fly and change the incoming request by appending header. I would recommend to check all the steps in my tutorial and github repo.
Here is my tutorial : https://medium.com/#java.developer.raman/enable-spring-config-server-to-use-cert-authentication-with-vault-as-back-end-ff84e1ef2de7?sk=45a26d7f1277437d91a5cff3d5997287
And GitHub repository: https://github.com/java-developer-raman/config-server-vault-backend
After reading the documentation on Spring cloud Zuul I have understood that with SEMAPHORE isolation and max concurrent requests of around 2, Zuul server can easily handle about 5000 rps. With a value of around 2000 it's throwing the following exception and showing error rate 100% when I tried to invoke the service from Jmeter.
com.netflix.hystrix.exception.HystrixRuntimeException:
Service1 could not acquire a semaphore for execution and no fallback available.
Then I have bumped up the number of max concurrent requests to 200000 but still it's throwing the exception but the error rate has gone down to 10%.
Can you please let me know the reason for this. Is this because of slow micro service or any configuration issue in Spring cloud Zuul ? The following is the configuration:
ribbon:
ConnectTimeout: 20000000
ReadTimeout: 20000000
MaxTotalHttpConnections: 5000
MaxHttpConnectionsPerHost: 5000
ActiveConnectionsLimit: 4000
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds: 20000000
zuul.hystrix.command.default.execution.isolation.strategy: SEMAPHORE
zuul.hystrix.command.default.execution.isolation.semaphore.maxConcurrentRequests: 2000
zuul.hystrix.command.default.fallback.isolation.semaphore.maxConcurrentRequests: 2000
zuul.eureka.default.semaphore.maxSemaphores: 30000
After some tests, I found it should be:
zuul.semaphore.maxSemaphores: 30000
It is different from the issue in github. Maybe it is related with the version.
I use Eureka as my service discovery and as a load balancer, and it is working fine when having two instances of a service "A", however when I stop one of this instances Eureka doesn't recognize that one of the instances is down and it keeps me showing an error page everytime the load balancer tries to use the dead instance.
I have put the enableSelfPreservation to false to prevent that but it takes Eureka up to 3 - 5 minutes to unregister that service, however I want high availability over my service and I want to perform the failover immediately in a matter of seconds. Is this possible using Eureka, if not how can I achieve to use only the alive instances when the others are dead.
I am using spring boot, here is my configuration for the Eureka server.
server:
port: 8761
eureka:
instance:
hostname: localhost
client:
registerWithEureka: false
fetchRegistry: false
serviceUrl:
defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/
server:
enableSelfPreservation: false
You should add a ribbon configuration to your application.yml. It is also recommended to set the hystrix isolation level to THREAD with a timeout set.
Note: This configuration should be in the client side (this usually means your gateway server), since Ribbon (and Spring Cloud in general) are a form of client-side load balancing.
Here's an example that I use:
hystrix:
command:
default:
execution:
isolation:
strategy: THREAD
thread:
timeoutInMilliseconds: 40000 #Timeout after this time in milliseconds
ribbon:
ConnectTimeout: 5000 #try to connect to the endpoint for 5 seconds.
ReadTimeout: 5000 #try to get a response after successfull connection for 5 seconds
# Max number of retries on the same server (excluding the first try)
maxAutoRetries: 1
# Max number of next servers to retry (excluding the first server)
MaxAutoRetriesNextServer: 2