OPA Gatekeeper and issue while doing a cluster restore
Posted on December 5, 2022 by Adrian Wyssmann ‐ 5 min read
We recently encountered a huge problem, when using OPA Gatekeeper in a Rancher cluster and performing a restore of this cluster
Issues with cluster restore and Gatekeeper
It has been some weeks since I have installed Gatekeeper. Today we had an issue, which lead me to do a restore of the cluster state to the state from some minutes ago. Usually that is not a problem - well at least before OPA Gatekeeper was installed. While the restore worked, the cluster eventually will not start, some pods stuck in Termination while others are stuck in Pending:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
.........
cattle-gatekeeper-system pod/gatekeeper-controller-manager-87d467bdc-7s76h 1/1 Terminating 0 19d 10.32.13.196 server07 <none> <none>
cattle-gatekeeper-system pod/gatekeeper-controller-manager-87d467bdc-v565l 1/1 Terminating 0 19d 10.32.8.68 server08 <none> <none>
cattle-monitoring-system pod/pushprox-kube-controller-manager-proxy-7c446ddb95-tbwtl 0/1 Terminating 0 5m17s <none> server07 <none> <none>
credit-agreement pod/credit-agreement-frontend-web-86bf44557c-9dwvz 2/2 Terminating 0 103d 10.32.8.154 server08 <none> <none>
ingress-nginx pod/nginx-ingress-controller-79dxf 0/1 Terminating 0 147m <none> server07 <none> <none>
ingress-nginx pod/nginx-ingress-controller-wf8k4 0/1 Terminating 0 148m <none> server08 <none> <none>
......
kube-system pod/canal-g7n6d 0/2 Pending 0 5m4s <none> server08 <none> <none>
kube-system pod/canal-zpxjl -1/2 Pending 0 5m4s <none> server07 <none> <none>
kube-system pod/coredns-b3d8fc84d-c7qrr 0/1 Pending 0 168m <none> server07 <none> <none>
kube-system pod/metrics-server-66b8c99467-tdpk6 0/1 Terminating 0 89d 10.32.8.29 server08 <none> <none>
We can see the events in the namespace cattle-system:
$ kubectl get event -n kube-system --sort-by='.lastTimestamp'
...
Reason Resource Date FailedMount | Pod istio-cni-node-b8v6tMountVolume.SetUp failed for volume "kube-api-access-dcqs8" : failed to fetch token: pods "istio-cni-node-b8v6t" not found | Tue, Nov 22 2022 6:31:28 pm
FailedMount | Pod canal-6gx6zMountVolume.SetUp failed for volume "kube-api-access-7xdmz" : failed to fetch token: pods "canal-6gx6z" not found | Tue, Nov 22 2022 6:31:24 pm
FailedMount | Pod nginx-ingress-controller-bp47jMountVolume.SetUp failed for volume "kube-api-access-dv9z6" : failed to fetch token: pods "nginx-ingress-controller-bp47j" not found | Tue, Nov 22 2022 6:30:54 pm
FailedMount | Pod istio-cni-node-b82kqMountVolume.SetUp failed for volume "kube-api-access-xz5rm" : failed to fetch token: pods "istio-cni-node-b82kq" not found | Tue, Nov 22 2022 6:30:53 pm
FailedMount | Pod coredns-autoscaler-d958447b4-95zwbMountVolume.SetUp failed for volume "kube-api-access-g88jq" : failed to fetch token: pods "coredns-autoscaler-d958447b4-95zwb" not found | Tue, Nov 22 2022 6:30:47 pm
...
Furthermore, we can also see these errors in the logs:
E1121 18:32:50.705702 1 leaderelection.go:367] Failed to update lock: Put "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1122 18:32:58.329994 1 leaderelection.go:367] Failed to update lock: the server was unable to return a response in the time allotted, but may still be processing the request (put leases.coordination.k8s.io kube-scheduler)
We eventually came out of this missery by deleting the Gatekeeper webhooks
$ kubectl delete ValidatingWebhookConfiguration gatekeeper-validating-webhook-configuration
validatingwebhookconfiguration.admissionregistration.k8s.io "gatekeeper-validating-webhook-configuration" deleted
$ kubectl delete MutatingWebhookConfiguration gatekeeper-mutating-webhook-configuration
mutatingwebhookconfiguration.admissionregistration.k8s.io "gatekeeper-mutating-webhook-configuration" deleted
After that we also had to restart some of the nodes, on which the pods were on state Pending.
Configure Gatekeeper properly
When looking at Exempting Namespaces it’s important to notice this statement:
If you use
--exempt-namespaceflag andadmission.gatekeeper.sh/ignorelabel, Gatekeeper’s webhook will not be called by the API server for any resource in that namespace. That means that Gatekeeper being down should have no effect on requests for that namespace.
When looking at the helm chart, more specificall at gatekeeper-controller-manager-deployment.yaml we can see that one has to provide exemptNamespaces and exemptNamespacePrefixes:
...
{{- range .Values.controllerManager.exemptNamespaces}}
- --exempt-namespace={{ . }}
{{- end }}
{{- range .Values.controllerManager.exemptNamespacePrefixes}}
- --exempt-namespace-prefix={{ . }}
{{- end }}
...
Hence the values file has to be extended to your needs, especially under exemptNamespace you may need to add additional namespaces:
...
controllerManager:
...
exemptNamespace:
- ingress-nginx
- istio-system
- ...
exemptNamespacePrefixes:
- "cattle"
- "kube"
...
Before continue, let’s add back the webhooks we deleted above:
$ kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io -A
NAME WEBHOOKS AGE
gatekeeper-validating-webhook-configuration 2 6s
ingress-nginx-admission 1 453d
istio-validator-istio-system 1 214d
istiod-default-validator 1 106d
rancher-monitoring-admission 1 453d
$ kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io -A
NAME WEBHOOKS AGE
gatekeeper-mutating-webhook-configuration 1 40s
istio-revision-tag-default 4 4d5h
istio-sidecar-injector 4 214d
rancher-monitoring-admission 1 453d
vpa-webhook-config 1 4m39s
At last, we have to add the missing label to all namespaces. I run this script:
#!/usr/bin/env bash
projectId=$(kubectl get ns cattle-system -o jsonpath='{.metadata.labels.field\.cattle\.io/projectId}')
for ns in $(kubectl get ns --selector field.cattle.io/projectId=$projectId -o name); do
echo "$ns ($CONTEXT):"
if [[ -z "$CONTEXT" ]]; then
kubectl label $ns admission.gatekeeper.sh/ignore="true"
else
kubectl label $ns admission.gatekeeper.sh/ignore="true" --context $CONTEXT
fi
done
Note
Ensure that the namespaces are exempt, before you run the script above, otherwise you get this error
Error from server (Only exempt namespace can have the admission.gatekeeper.sh/ignore label): admission webhook "check-ignore-label.gatekeeper.sh" denied the request: Only exempt namespace can have the admission.gatekeeper.sh/ignore label
After all is in place, we create a new snapshot which we then will restore. Unfortunately, we still have some pods which are in state Pending and Terminating:
$ kubectl get pods \
--field-selector="status.phase!=Succeeded,status.phase!=Running" -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cattle-system cattle-cluster-agent-58585d647d-4w2ww 0/1 Pending 0 52s
cattle-system cattle-cluster-agent-58585d647d-xx5rq 0/1 Pending 0 52s
kube-system canal-6qswk 0/2 Pending 0 52s
kube-system canal-gjkcr 0/2 Pending 0 52s
kube-system canal-m9m94 0/2 Pending 0 52s
kube-system coredns-7ddc8976bb-kjxql 0/1 Pending 0 52s
kube-system coredns-autoscaler-9988dc68-72zkw 0/1 Pending 0 52s
kube-system metrics-server-65b8c99467-p45ns 0/1 Pending 0 52s
Still we observe the same events in the namespace kube-system …
$ kubectl get event -n kube-system --sort-by='.lastTimestamp'
...
2m34s Normal SuccessfulCreate replicaset/coredns-autoscaler-9988dc68 Created pod: coredns-autoscaler-9988dc68-72zkw
2m34s Warning Unhealthy pod/canal-vx4nl Readiness probe failed: Get "http://localhost:9099/readiness": dial tcp 127.0.0.1:9099: connect: connectio
n refused
2m34s Warning FailedKillPod pod/calico-kube-controllers-7586d58b74-l9dzl error killing pod: failed to "KillPodSandbox" for "32f3dd48-96ef-44e8-992a-afd78356164c" with KillPodSandb
oxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"calico-kube-controllers-7586d58b74-l9dzl_kube-system\" network: cni config uninitialized"
2m34s Normal SuccessfulCreate replicaset/calico-kube-controllers-7586d58b74 Created pod: calico-kube-controllers-7586d58b74-6qt4d
2m34s Warning Unhealthy pod/canal-vx4nl Liveness probe errored: rpc error: code = Unknown desc = container not running (dc23c53f0eb7b56567c4116283
0ffbeefa8b3d782e1d00a5503860524319509b)
...
40s Normal Killing pod/metrics-server-65b8c99467-28bxh Stopping container metrics-server
11s Warning FailedKillPod pod/metrics-server-65b8c99467-28bxh error killing pod: failed to "KillPodSandbox" for "a3adecab-f767-4f2d-9100-eff5a2fbc4da" with KillPodSandb
oxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"metrics-server-65b8c99467-28bxh_kube-system\" network: error getting ClusterInformation: connection is unauthorized: Unaut
horized"
...
… and in namespace cattle-system:
$ kubectl get event -n cattle-system
...
6m46s Warning FailedKillPod pod/helm-operation-cm2kb error killing pod: failed to "KillPodSandbox" for "ee233c56-a656-429b-8fa7-ce0affb19556" with KillPodSandboxError: "rp
c error: code = Unknown desc = networkPlugin cni failed to teardown pod \"helm-operation-cm2kb_cattle-system\" network: cni config uninitialized"
...
Ultimately, it helped to restart the nodes, on which the pods were in Pending state.
Conclusion
When using Gatekeeper, it is essential to have important namespaces exempt and add label admission.gatekeeper.sh/ignore to all namespaces in the system project.