#5 - kubectl plugins

★ Any command in your $PATH that is prefixed with kubectl- becomes a kubectl plugin

★ Krew - package manager for kubectl plugins - github.com/kubernetes-sigs/krew

★ Install plugins with: kubectl krew install <PLUGIN NAME>

★ Some of my fave plugins: ○ stern - Multi-pod/container log tailing ○ tree - Show hierarchy of resources based on ownerReferences ○ outdated - Find containers with outdated images ○ gs - Giant Swarm’s plugin for working with our managed clusters

Plugins can be in any language. You can easily add your own by creating Bash scripts with a kubectl- prefixed name. Note: autocomplete is a bit trickier here. Some plugins support it but generally expect your tabcompletion to only recommend core kubectl features.

#6 - kshell / kubectl debug

Debugging a running pod - kubectl exec

kubectl exec my-broken-pod -it — sh

error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec……

Debugging a running pod - kubectl debug (Requires Kubernetes v1.23)

kubectl debug -it —image bash my-broken-pod

Defaulting debug container name to debugger-gprmk. If you don’t see a command prompt, try pressing enter. bash-5.1#

kubectl exec is great for debugging misconfigured pods that aren’t crashing and have enough OS to exec into. But… If the pod is CrashLooping you’ll get kicked out of the session when it crashes. If the pod doesn’t have a shell you can exec into (e.g. a container that only has a Golang binary) you’ll not be able to exec kubectl debug is great for pods that either don’t have any OS

#6 - kshell / kubectl debug

Example - investigate a CrashLooping pod

kubectl run debug-demo —image=bash — exit 1

kubectl get pods debug-demo

NAME debug-demo READY 0/1 STATUS CrashLoopBackOff RESTARTS 2 (20s ago) AGE 44s

kubectl debug -it —image bash debug-demo

Defaulting debug container name to debugger-5mkjj. If you don’t see a command prompt, try pressing enter. bash-5.1#

kubectl debug has a few different modes: - launches an “ephemeral container” within the pod you’re debugging - kubectl debug - creates a copy of the pod with some values replaced (e.g. the image used) - kubectl debug –copy-to - launch a pod in the nodes host namespace to debug the node - kubectl debug node/my-node This has some limitations - cannot access all filesystem of failing container, only volumes that are shared

#7 - kube-ssh ★ github.com/AverageMarcus/kube-ssh (or github.com/giantswarm/kubectl-enter) ★ Give ssh-like access to a node’s underlying host, great for instances where nodes are provisioned without SSH or access is blocked. sh -c “$(curl -sSL https://raw.githubusercontent.com/AverageMarcus/kube-ssh/master/ssh.sh)” [0] [1] [2] Which ip-10-18-21-146.eu-west-1.compute.internal ip-10-18-21-234.eu-west-1.compute.internal ip-10-18-21-96.eu-west-1.compute.internal node would you like to connect to? 1 Please verify any script before you execute it! ⤴ If you don’t see a command prompt, try pressing enter. [root@ip-10-18-21-234 ~]# Why? - I prefer to use ephemeral instances with the minimal needed to run Kubernetes, no sshd, no port 22 open etc. but there are times when you just need to check what’s actually going on with the underlying host machine. Always verify a shell script before you run it! Ideally, download it first and run that instead. Why? Smaller potential attack surface. Less chance of “hotfixes” or “tweaks” being forgotten about.

#7 - kube-ssh How it works ssh.sh Dockerfile kubectl run kube-ssh —restart=Never -it —rm —image overridden —overrides ’ { “spec”: { “hostPID”: true, “hostNetwork”: true, ‘”${NODE_SELECTOR}”’ Ensure we can run on any node “tolerations”: [{ “operator”: “Exists” }], “containers”: [ { “name”: “kube-ssh”, Container image containing nsenter “image”: “averagemarcus/kube-ssh:latest”, “stdin”: true, “tty”: true, Allows us to switch to a host PID “securityContext”: { “privileged”: true } } ] } }’ —attach “$@” ⃔ ⃔ ⃔ FROM debian:buster as builder WORKDIR /tmp RUN apt-get update && \ apt-get install -yq \ make gcc gettext autopoint \ bison libtool automake pkg-config ADD https://github.com/karelzak/util-linux/archive/v2.34.tar.gz . RUN tar -xf v2.34.tar.gz && \ mv util-linux-2.34 util-linux \ cd util-linux && \ ./autogen.sh && \ ./configure && \ make LDFLAGS=”—static” nsenter FROM scratch COPY —from=builder /tmp/util-linux/nsenter / ENTRYPOINT [ “/nsenter”, “—all”, “—target=1”, “—”, “su”, “-” ] On the left: a snippet of the ssh.sh script that launches a new pod on the desired node (not shown here is the node selection logic) On the right: the Dockerfile of the image that is used for the pod. It just builds nsenter and uses it to switch to the host scope

#8 - Webhooks ⃔ OK, actually 3 but we’re ignoring CRD conversion webhooks ★ Two types of webhooks: ValidatingWebhook Ability to block actions against the API server if fails to meet given criteria. MutatingWebhook Modify requests before passing them on to the API server. ★ Implement more advanced access control than is possible with RBAC. [Restricting cluster-admin permissions] ★ Add default labels to resources as they’re created. ★ Enforce policies such as not using latest as an image tag or ensuring all workloads have resource requests/limits specified. ★ “Hotfix” for security issues (e.g. mutating all pods to include a LOG4J_FORMAT_MSG_NO_LOOKUPS env var to prevent Log4Shell exploit). [Log4Shell Mitigation] Allows for subtractive access control (take away a users ability to perform a certain action against a certain resource) - something not possible with RBAC See blog post about how we avoided a nasty bug in our CLI tool with a ValidatingWebhook. https://www.giantswarm.io/blog/restricting-cluster-admin-permissions

#8 - Webhooks ★ Build your own operator to implement custom logic ★ Kyverno - Kubernetes native policy management. Create Policy and ClusterPolicy resources to define rules in YAML ★ OPA Gatekeeper - Policy management built on top of Open Policy Agent Kyverno Policy apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: block-bulk-certconfigs-delete annotations: policies.kyverno.io/description: Block delete all bug in CLI spec: rules: - name: block-bulk-certconfigs-delete match: any: - resources: kinds: [CertConfig] preconditions: any: - key: “” operator: Equals value: “” validate: message: | Your current kubectl-gs version contains a critical bug deny: conditions: - key: “” operator: In value: [DELETE] Taken from our Restricting cluster-admin permissions blog post ⤴ Three most common approaches to leverage webhooks

#8 - Webhooks Notes: ⃔ This is one of the main causes we see of clusters being down ★ Where possible always avoid applying webhooks to resources in kube-system. This can cause a deadlock if those pods try to come up before the webhook service is available. ★ Be aware of the failurePolicy property - it defaults to “fail” which can cause troubles if your service handling the webhook goes down. ★ The reinvocationPolicy property can be set if changes made by a MutatingWebhook may need to go through other defined webhooks again. ★ Ordering - first MutatingWebhooks then ValidatingWebhooks. No guaranteed control of order within these two phases. Webhooks can break a cluster. Make sure your service is resilient and that your webhooks don’t block critical workloads. Webhooks can be backed by either services within the cluster or pointing to an URL outside of the cluster.