I am running on kubernetes job (job-1) from base pod. It works for basic use case. For second use case, I want trigger another kubernetes job(job-2) from already running job: job-1. While running job-2 I get service account error as given below:
Error occurred while starting container for Prowler due to exception : Failure executing: POST at: https://172.20.0.1/apis/batch/v1/namespaces/my-namespace/jobs. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. jobs.batch is forbidden: User "system:serviceaccount:my-namespace:default" cannot create resource "jobs" in API group "batch" in the namespace "my-namespace".
I have created service account with required permissions as given below:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: my-sa-service-role-binding
subjects:
- kind: ServiceAccount
name: my-sa
namespace: my-namespace
roleRef:
kind: Role
name: my-namespace
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: my-sa-service-role
rules:
- apiGroups: [""]
resources: ["secrets", "pods"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["batch", "extensions"]
resources: ["jobs"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-sa
I am passing "my-sa" as service account name but still, it refers to default service account.
I am using fabric8io kubernetes client to trigger the job and below is my code:
final Job job = new JobBuilder()
.withApiVersion("batch/v1")
.withNewMetadata()
.withName("demo")
.withLabels(Collections.singletonMap("label1", "maximum-length-of-63-characters"))
.withAnnotations(Collections.singletonMap("annotation1", "some-annotation"))
.endMetadata()
.withNewSpec().withParallelism(1)
.withNewTemplate()
.withNewSpec().withServiceAccount("my-sa")
.addNewContainer()
.withName("prowler")
.withImage("demo-image")
.withEnv(env)
.endContainer()
.withRestartPolicy("Never")
.endSpec()
.endTemplate()
.endSpec()
.build();
If you see the error message in detail, you'll find that your client is not using the service account you created (my-sa). Its using the default service account in the namespace instead:
"system:serviceaccount:my-namespace:default" cannot create resource "jobs"
And it should be safe to assume, that the default service account will not be having the privileges to create jobs.
It should be worthwhile to look into the official documentation of fabric8io, to see how you can authenticate with a custom service-account. From what I could find in the docs, it should be mostly handled by mounting the secret, corresponding to the service-account into the pod, then configuring your application code or probably setting up an specific environment variable.
Related
I have created a secret in Kubernetes and mounted it in a volume, so part of the yaml file looks like this:
volumeMounts:
- name: test-key
readOnly: true
mountPath: /opt/key
Then the secret itself contains:
My problem comes when trying to retrieve it using in Spring. How I'm supposed to do it?
What I've tried so far is setting in the application.properties --> spring.datasource.private-key=${PRIVATE_KEY} but it's not working. It gives me a placeholder error:
Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'spring.datasource.private-key' in value "${spring.datasource.private-key}"
Any idea what I'm doing wrong?
UPDATE:
This is the way that Im reading the secret in spring, but it still give me the same error related to the placeholder when deploying:
#Value("${spring.datasource.private-key}")
private String privateKey;
#Bean
public PrivateKey getPrivateKeyFromEnvironmentVariable() throws IOException, NoSuchAlgorithmException {
List<String> activeProfiles = Arrays.asList(environment.getActiveProfiles());
String key;
if(activeProfiles.contains(LOCAL_ENVIRONMENT_NAME) ) {
key = resourceUtil.asString(LOCAL_PRIVATE_KEY_RESOURCE_PATH);
} else if(activeProfiles.contains(TEST_ENVIRONMENT_NAME)) {
key = generatePrivateKey();
} else {
//key = System.getenv(PRIVATE_KEY_ENVIRONMENT_VARIABLE_NAME);
key = privateKey;
}
I have created a secret in Kubernetes and mounted it in a volume.
But the sprint boot expects environment variable for the PRIVATE_KEY. So only binding will not help.
so I will suggest to create secrets
apiVersion: v1
data:
private_key: YWRtaW4=
kind: Secret
metadata:
name: mysecret
type: Opaque
Now reference this secret and set as an environment variable so sprintboot can understand.
apiVersion: v1
kind: Pod
metadata:
name: secret-env-pod
spec:
containers:
- name: mycontainer
image: busybox
command: ["env"]
env:
- name: PRIVATE_KEY
valueFrom:
secretKeyRef:
name: mysecret
key: private_key
if you run this, there will be an env name PRIVATE_KEY with value admin that is base64 YWRtaW4= in the secret file.
Spring Cloud Gateway has a ForwardRoutingFilter
The ForwardRoutingFilter looks for a URI in the exchange attribute ServerWebExchangeUtils.GATEWAY_REQUEST_URL_ATTR. If the URL has a forward scheme (such as forward:///localendpoint), it uses the Spring DispatcherHandler to handle the request.
I tried the following
exchange.getAttributes().put(GATEWAY_REQUEST_URL_ATTR, URI.create("forward:///debug"));
return chain.filter(exchange);
But it seems to still be routing to the original service rather than my endpoint
The routes are defined by discovery as follows
spring:
cloud:
gateway:
discovery:
locator:
enabled: true
predicates:
- name: Path
args:
patterns: "metadata['path']"
filters:
- RemoveRequestHeader='Cookie'
# Prevent propagation of traces
- RemoveRequestHeader='X-B3-TraceId'
- RemoveRequestHeader='X-Trace-ID'
- name: Retry
args:
retries: 3
methods:
- GET
- name: RewritePath
args:
regexp: "metadata['path.regexp']"
replacement: "metadata['path.replacement']"
- name: CircuitBreaker
args:
name: "'resilience'"
fallbackUri: "'forward:/unavailable'"
# The Grpc is the one that contains the filter
- name: Grpc
args:
serviceId: serviceId
host: host
useGrpc: "metadata['protocol'] == 'grpc'"
port: port
I am using the spring gateway implementation "org.springframework.cloud:spring-cloud-starter-gateway" RequestRateLimiter to limit the request rate, this is the RequestRateLimiter config:
- filters:
- name: JwtAuthentication
- name: RequestRateLimiter
args:
redis-rate-limiter.replenishRate: 1
redis-rate-limiter.burstCapacity: 2
id: time-capsule-service
order: 2
predicates:
- Path=/tik/**
uri: http://10.111.149.10:11015
Now I am facing a problem that all the requests return 403 http code. I am tracing the source code and found the key resolver gets an empty key. Must I customize the key by myself? BTW, I have already configured the redis server address.
I'm reading traffic management documentation and also use Istio for chaos testing.
I know we can use some headers value to routing traffic for AB testing, but what I would like to know is if I can do the same to return an error in one service or make a delay, as the Istio chaos testing provides.
So I can end up making request with header request-type:chaos, and for those we will apply the chaos testing YAML config, but not for the rest of request.
The RouteRule (used in the second link you sent) is deprecated and replaced by VirtualService:
VirtualService, DestinationRule, and ServiceEntry replace RouteRule, DestinationPolicy, and EgressRule respectively.
Good instructions on how to configure an HTTP delay fault is presented in this -> Fault Injection documentation.
Example YAML config for your header (request-type:chaos) based on Ingress Gateways and Fault Injection documentations:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- "httpbin.example.com"
gateways:
- httpbin-gateway
http:
- fault:
delay:
fixedDelay: 7s
percent: 100
match:
- headers:
request-type:
exact: chaos
route:
- destination:
port:
number: 80
host: my-echo-service
- route:
- destination:
port:
number: 80
host: my-nginx-service
EDIT: Should work as well for apiVersion: networking.istio.io/v1beta1.
I have web app that allow sent messages to queue, it deployed on Websphere Application Server and work very well.
I try to build light environment for autotests, but when i try to sent message to queue from test it returns to me MQJE001: Completion Code '2', Reason '2035'
I thought that problem in CHLAUTH rules but seems that i have all rights.
C:/> dspmqaut -m M00.EDOGO -n OEP.FROM.GW_SBAST.DLV -t q -p out-bychek-ao
Entity out-bychek-ao has the following authorizations for object OEP.FROM.GW_SBA
ST.DLV:
get
browse
put
inq
set
crt
dlt
chg
dsp
passid
passall
setid
setall
clr
error from logs :
AMQ8075: Authorization failed because the SID for entity 'out-bychek-a' cannot
be obtained.
EXPLANATION:
The Object Authority Manager was unable to obtain a SID for the specified
entity. This could be because the local machine is not in the domain to locate
the entity, or because the entity does not exist.
ACTION:
Ensure that the entity is valid, and that all necessary domain controllers are
available. This might mean creating the entity on the local machine.
----- amqzfubn.c : 2252 -------------------------------------------------------
7/9/2018 15:39:57 - Process(2028.3) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(SBT-ORSEDG-204) Installation(Installation1)
VRMF(7.5.0.4) QMgr(M00.EDOGO)
AMQ9557: Queue Manager User ID initialization failed.
EXPLANATION:
The call to initialize the User ID failed with CompCode 2 and Reason 2035.
ACTION:
Correct the error and try again.
----- cmqxrsrv.c : 1975 -------------------------------------------------------
7/9/2018 15:39:57 - Process(2028.3) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(SBT-ORSEDG-204) Installation(Installation1)
VRMF(7.5.0.4) QMgr(M00.EDOGO)
AMQ9999: Channel 'SC.EDOGO' to host '10.82.38.188' ended abnormally.
EXPLANATION:
The channel program running under process ID 2028(11564) for channel 'SC.EDOGO'
ended abnormally. The host name is '10.82.38.188'; in some cases the host name
cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
----- amqrmrsa.c : 909 --------------------------------------------------------
notice AMQ8075: Authorization failed because the SID for entity 'out-bychek-a' cannot in my account name lost last letter. Is it normal?
and this
DISPLAY CHLAUTH('SYSTEM.DEF.SVRCONN') MATCH(RUNCHECK) ALL ADDRESS('127.0.0.1') CLNTUSER('out-bychek-ao')
7 : DISPLAY CHLAUTH('SYSTEM.DEF.SVRCONN') MATCH(RUNCHECK) ALL ADDRESS('127.0.0.1') CLNTUSER('out-bychek-ao')
AMQ8898: Display channel authentication record details - currently disabled.
CHLAUTH(SYSTEM.*) TYPE(ADDRESSMAP)
DESCR(Default rule to disable all SYSTEM channels)
CUSTOM( ) ADDRESS(*)
USERSRC(NOACCESS) WARN(NO)
ALTDATE(2016-11-14) ALTTIME(17.33.34)
dmpmqaut -m M00.EDOGO -n OEP.FROM.GW_SBAST.DLV -t q -p out-bychek-ao -e
profile : OEP.FROM.GW_SBAST.DLV
object type: queue
entity : out-bychek-ao#alpha
entity tyoe: principal
authority : allmqi dlt chg dsp clr
- - - - - - - - -
profile : CLASS
object type: queue
entity : out-bychek-ao#alpha
entity tyoe: principal
authority : clt