Develop Reference

Istio routing to chaos testing

I'm reading traffic management documentation and also use Istio for chaos testing.
I know we can use some headers value to routing traffic for AB testing, but what I would like to know is if I can do the same to return an error in one service or make a delay, as the Istio chaos testing provides.
So I can end up making request with header request-type:chaos, and for those we will apply the chaos testing YAML config, but not for the rest of request.

The RouteRule (used in the second link you sent) is deprecated and replaced by VirtualService:
VirtualService, DestinationRule, and ServiceEntry replace RouteRule, DestinationPolicy, and EgressRule respectively.
Good instructions on how to configure an HTTP delay fault is presented in this -> Fault Injection documentation.
Example YAML config for your header (request-type:chaos) based on Ingress Gateways and Fault Injection documentations:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- "httpbin.example.com"
gateways:
- httpbin-gateway
http:
- fault:
delay:
fixedDelay: 7s
percent: 100
match:
- headers:
request-type:
exact: chaos
route:
- destination:
port:
number: 80
host: my-echo-service
- route:
- destination:
port:
number: 80
host: my-nginx-service
EDIT: Should work as well for apiVersion: networking.istio.io/v1beta1.

OpenFeign + Hystrix - Different timeout for different clients

I have Hystrix working with Feign in my Spring Boot Application.
I have set the default timeout for Hystrix to 10000ms with:
feign:
hystrix:
enabled: true
hystrix:
command:
default:
execution:
isolation:
thread:
timeoutInMilliseconds: 10000
The problem is I have this one client, lets call it HeavyClient, that is a heavy call that sometimes takes more time and causes the circuit break.
I would like to increase the timeout cap in Hystrix for this one guy only. Is it possible?
I have tried with Feign properties like:
feign:
client:
config:
HeavyClient:
connectTimeout: 30000
readTimeout: 30000
But this doesn't work. I guess Hystrix does not check Feign properties.
I'm using:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
<version>2.2.1.RELEASE</version>
</dependency>
Any help is appreciated, thanks!

After tweeking around and search more deeply I have found on this GitHub issue that default naming for Hystrix commands are something ugly like InterfaceName#methodNameSignature().
So, for instance, given the following #FeignClient:
#FeignClient(name = "heavyClient", url = "${heavyclient.url}")
public interface HeavyClient {
#RequestMapping("/process/{operation}")
public ResponseEntity<Response> process(#PathVariable("operation") String operation);
}
It would be configured as:
hystrix:
command:
default:
execution:
isolation:
thread:
timeoutInMilliseconds: 5000
HeavyClient#process(String):
execution:
isolation:
thread:
timeoutInMilliseconds: 30000
Unfortunelly, this did not work for me... Not sure why... =s
So to solve it, I registered a bean of type SetterFactory that would create the command keys given the #FeignClient name:
#Bean
public SetterFactory setterFactory() {
return (target, method) -> HystrixCommand.Setter
.withGroupKey(HystrixCommandGroupKey.Factory.asKey(target.name()))
.andCommandKey(HystrixCommandKey.Factory.asKey(target.name()));
}
And then I can use the configuration simply like:
hystrix:
command:
heavyClient:
execution:
isolation:
thread:
timeoutInMilliseconds: 30000

CircuitBreakerNameResolver to naming circuit as HystrixCommandKey, there is a default implementation DefaultCircuitBreakerNameResolver, which output key pattern is HardCodedTarget#methodName(ParamClass), so you can customize CircuitBreakerNameResolver instead of DefaultCircuitBreakerNameResolver.

MQ7 with Java 7 and SSL is not working., it was working before 6 months

We have One QM and One CHANNEL and many QUEUES created for clients. Around 5 clients are connected to this QM for their transactions. Each 5 clients connected to their respective QUEUES . There is a jks file created in this QM for SSL connection. Each 5 clients connect with jks file + SSL_RSA_WITH_RC4_128_SHA from their javaClient. QM is also configured with SSLCIPH(RC4_SHA_US).
Now all of a sudden , without any javaClient change , 1 client could not able to connect to configured QM. All others are able to connect to same QM , without any issue.
AMQERR01.LOG is not logged with any specific exception or error
In application logs its saying common MQ exception
Error as com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2397'
2397 - Cipher spec<>suite not matching--is any possibility?
we enabled tracing (strmqtrc -m TEST.QM -t detail -t all) and saw Trace logs in path (C:\Program Files (x86)\IBM\Websphere MQ\trace) ,but could not get any details on why SSL-connection could not happening?
We done one more exercise like created a new QM for issue client and tested without SSL and its working. When we enabled SSL in new QM and javaClient , the same 2397 started logging.
Could someone guide me for better logging and tracing in MQ , which can see why 2397 is throwing?
Could someone guide me for better logging and tracing in Java using -D [-Djavax.net.debug=all] , which can see why 2397 is throwing?
MQ Version ->7
MQ Server in ->Windows
from trace logs
returning TEST.QM
Freeing cbmindex:0 pointer:24DDB540 length:2080
-----} TreeNode.getMQQmgrExtObject (rc=OK)
cbmindex:10
-------------} xcsFreeMemFn (rc=OK)
------------} amqjxcoa.wmqGetAttrs (rc=OK)
-----{ UiQueueManager.testQmgrAttribute
-------------{ Message.getMessage
testing object 'TEST.QM'
An internal method detected an unexpected system return code. The method {0} returned {1}. (AMQ4580)
checking attribute 'QmgrCmdLevelGreaterThan'
-------------} Message.getMessage (rc=OK)
for value '510'
-----------}! NativeCalls.getAttrs (rc=Unknown(C35E))
-----} UiQueueManager.testQmgrAttribute (rc=OK)
Message = An internal method detected an unexpected system return code. The method wmq_get_attrs returned "retval.rc2 = 268460388". (AMQ4580), msgID = AMQ4580, rc = 50014, reason = 268460388, severity = 30
result = true
---} TreeNode.testAttribute (rc=OK)
---{ TreeNode.testAttribute
-----{ QueueManagerTreeNode.toString
-----} QueueManagerTreeNode.toString (rc=OK)
testing object 'TEST.QM'
checking attribute 'OamTreeNode'
-----------{ NativeCalls.getAttrs
------------{ amqjxcoa.wmqGetAttrs
qmgr:2A7B32C8, stanza:2A7B32C4, version:1
for value 'true'
QMgrName('TEST.QM')
-----{ TreeNode.getMQQmgrExtObject
StanzaName('QMErrorLog')
testing object 'TEST.QM'
Full QM.INI filename: SOFTWARE\IBM\MQSeries\CurrentVersion\Configuration\QueueManager\TEST!QM, Multi-Instance: FALSE
--------------} xcsGetIniFilename (rc=OK)
--------------{ xcsGetIniAttrs
---------------{ xcsBrowseIniCallback
FileType = (1)
----------------{ xcsBrowseRegistryCallback
xcsBrowseRegistryCallback
-----------------{ xusAddStanzaLineList
------------------{ xcsGetMemFn
checking attribute 'PluginEnabled'
component:24 function:15 length:2080 options:0 cbmindex:0 *pointer:24DDB540
------------------} xcsGetMemFn (rc=OK)
for value 'com.ibm.mq.explorer.oam'
RetCode (OK)
-----------------} xusAddStanzaLineList (rc=OK)
-----------------{ xusAddStanzaLineList
------------------{ xcsGetMemFn
-----{ UiPlugin.isPluginEnabled
component:24 function:15 length:2080 options:0 cbmindex:1 *pointer:24DDDFE8
------------------} xcsGetMemFn (rc=OK)
RetCode (OK)
-----------------} xusAddStanzaLineList (rc=OK)
testing plugin_id: com.ibm.mq.explorer.oam
-----------------{ xurGetSpecificRegStanza
-------{ PluginRegistrationManager.isPluginEnabled
Couldn't open key (QMErrorLog) result 2: The system cannot find the file specified.
MQ version 7.0.1.9
jdk1.8.0_181-i586
com.ibm.mq*jar Version
Specification -version : 6.0.2.1
Implementation-Version :6.0.2.1 -j600-201-070305

Zuul: automatic rerouting incoming requests to other service instance in case of unavailable service

I have configured Zuul with Eureka in a way, that 3 identical instances of a service are working parallely. I am calling the gateway on the port 8400, which routes incoming requests to ports 8420, 8430 and 8440 in a round-robin manner. It works smoothly. Now, if I switching off one of the 3 services, a small amount of incoming requests will go wrong with the following exception:
com.netflix.zuul.exception.ZuulException: Filter threw Exception
=> 1: java.util.concurrent.FutureTask.report(FutureTask.java:122)
=> 3: hu.perit.spvitamin.core.batchprocessing.BatchProcessor.process(BatchProcessor.java:106)
caused by: com.netflix.zuul.exception.ZuulException: Filter threw Exception
=> 1: com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:227)
caused by: org.springframework.cloud.netflix.zuul.util.ZuulRuntimeException: com.netflix.zuul.exception.ZuulException: Forwarding error
=> 1: org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:124)
caused by: com.netflix.zuul.exception.ZuulException: Forwarding error
=> 1: org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.handleException(RibbonRoutingFilter.java:198)
caused by: com.netflix.client.ClientException: com.netflix.client.ClientException
=> 1: com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:118)
caused by: java.lang.RuntimeException: org.apache.http.NoHttpResponseException: scalable-service-2:8430 failed to respond
=> 1: rx.exceptions.Exceptions.propagate(Exceptions.java:57)
caused by: org.apache.http.NoHttpResponseException: scalable-service-2:8430 failed to respond
=> 1: org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
My Zuul routing looks like this:
### Zuul routes
zuul.routes.scalable-service.path=/scalable/**
#Authorization header will be forwarded to scalable-service
zuul.routes.scalable-service.sensitiveHeaders: Cookie,Set-Cookie
zuul.routes.scalable-service.serviceId=template-scalable-service
It takes a while until Eureka discovers the service is not available any more.
My question is: Is there a possibility, to configure Zuul so that in case of a NoHttpResponseException, it forwards the requests to another available instance in the pool?

Eureka, by default, requires lease to be renewed every 90s. That is, if a service instance doesn't get its lease renewed in 90s, Eureka server will evict the instance. In your case, the instance has not been evicted yet - the renew window for the instance was valid.
For this, you can decrease the renew duration through config setup at eureka client and eureka server as described here.
Note: If you hit the actuator /shutdown endpoint, the instance is immediately evicted

Finally I found the solution to the problem. The appropriate search phrase was 'fault tolerance'. The key is the autoretry config in the following application.properties file. The value of template-scalable-service.ribbon.MaxAutoRetriesNextServer must be set at least to 6 in case of 3 pooled services to achieve full fault tolerance. With that setup I can kill 2 of 3 services any time, no incoming request will go wrong. Finally I have set it to 10, there is no unnecessary increase of timeout, hystrix will break the line.
### Eureka config
eureka.instance.hostname=${hostname:localhost}
eureka.instance.instanceId=${eureka.instance.hostname}:${spring.application.name}:${server.port}
eureka.instance.non-secure-port-enabled=false
eureka.instance.secure-port-enabled=true
eureka.instance.secure-port=${server.port}
eureka.instance.lease-renewal-interval-in-seconds=5
eureka.instance.lease-expiration-duration-in-seconds=10
eureka.datacenter=perit.hu
eureka.environment=${EUREKA_ENVIRONMENT_PROFILE:dev}
eureka.client.serviceUrl.defaultZone=${EUREKA_SERVER:https://${server.fqdn}:${server.port}/eureka}
eureka.client.server.waitTimeInMsWhenSyncEmpty=0
eureka.client.registry-fetch-interval-seconds=5
eureka.dashboard.path=/gui
eureka.server.enable-self-preservation=false
eureka.server.expected-client-renewal-interval-seconds=10
eureka.server.eviction-interval-timer-in-ms=2000
### Ribbon
ribbon.IsSecure=true
ribbon.NFLoadBalancerPingInterval=5
ribbon.ConnectTimeout=30000
ribbon.ReadTimeout=120000
### Zuul config
zuul.host.connectTimeoutMillis=30000
zuul.host.socketTimeoutMillis=120000
zuul.host.maxTotalConnections=2000
zuul.host.maxPerRouteConnections=200
zuul.retryable=true
### Zuul routes
#template-scalable-service
zuul.routes.scalable-service.path=/scalable/**
#Authorization header will be forwarded to scalable-service
zuul.routes.scalable-service.sensitiveHeaders=Cookie,Set-Cookie
zuul.routes.scalable-service.serviceId=template-scalable-service
# Autoretry config for template-scalable-service
template-scalable-service.ribbon.MaxAutoRetries=0
template-scalable-service.ribbon.MaxAutoRetriesNextServer=10
template-scalable-service.ribbon.OkToRetryOnAllOperations=true
#template-auth-service
zuul.routes.auth-service.path=/auth/**
#Authorization header will be forwarded to scalable-service
zuul.routes.auth-service.sensitiveHeaders=Cookie,Set-Cookie
zuul.routes.auth-service.serviceId=template-auth-service
# Autoretry config for template-auth-service
template-auth-service.ribbon.MaxAutoRetries=0
template-auth-service.ribbon.MaxAutoRetriesNextServer=0
template-auth-service.ribbon.OkToRetryOnAllOperations=false
### Hystrix
hystrix.command.default.execution.timeout.enabled=false
Beside of this, I have a profile specific setup in application-discovery.properties
#Microservice environment
eureka.client.registerWithEureka=false
eureka.client.fetchRegistry=true
spring.cloud.loadbalancer.ribbon.enabled=true
I start my server in a docker container like this:
services:
discovery:
container_name: discovery
image: template-eureka
environment:
#agentlib for remote debugging
- JAVA_OPTS=-DEUREKA_SERVER=https://discovery:8400/eureka -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
- TEMPLATE_EUREKA_OPTS=-Dspring.profiles.active=default,dev,discovery
- EUREKA_ENVIRONMENT_PROFILE=dev
ports:
- '8400:8400'
- '5500:5005'
networks:
- back-tier-net
- monitoring
hostname: 'discovery'
See the complete solution in GitHub.

How to configure Ribbon/Hystrix per route or per endpoint using Zuul Proxy

I have an ft-admin microservive which exposes 2 endpoints app-config and app-analytic. In my #EnableZuulProxy gateway project, I can define the routing rules and specify other configurations for Ribbon and Hystrix on microservice-level using the serviceId as following.
zuul:
routes:
admin-services:
path: /admin/**
serviceId: ft-admin
stripPrefix: true
ft-admin:
ribbon:
ActiveConnectionsLimit: 2
hystrix:
command:
ft-admin:
execution:
isolation:
thread:
timeoutInMilliseconds: 10000
I'm wondering if there's a way to bring the above configurations down to endpoint-level for each app-config and app-analytic individually. The goal is to be able to give a different setting for each endpoint as following.
zuul:
routes:
app-config-endpoint:
path: /app-config/**
serviceId: ft-admin
stripPrefix: false
app-analytic-endpoint:
path: /app-analytic/**
serviceId: ft-admin
stripPrefix: false
app-config-endpoint:
ribbon:
ActiveConnectionsLimit: 5
app-analytic-endpoint:
ribbon:
ActiveConnectionsLimit: 2
...
When I run my gateway project in Debug mode with ft-admin.ribbon.ActiveConnectionsLimit: 2, I can see the following lines in the log.
c.netflix.config.ChainedDynamicProperty : Property changed: 'ft-admin.ribbon.ActiveConnectionsLimit = 2'
c.netflix.config.ChainedDynamicProperty : Flipping property: ft-admin.ribbon.ActiveConnectionsLimit to use it's current value:2
However, when I run my project with app-config-endpoint.ribbon.ActiveConnectionsLimit: 5, I see the following lines.
c.netflix.config.ChainedDynamicProperty : Property changed: 'ft-admin.ribbon.ActiveConnectionsLimit = -2147483648'
c.netflix.config.ChainedDynamicProperty : Flipping property: ft-admin.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
I've tried to search through a ton of posts but it seems like the configurations always stop at the microservice-level. The endpoint/route level configurations are completely ignore.
I'd be very grateful if you could point me in the right direction or tell me your story if you've tried this before.