Managing Kubernetes without losing your cool

A presentation at DDD East Midlands in October 2023 in Nottingham, UK by Marcus Noble

Hi 👋, I’m Marcus Noble, a platform engineer at I’m found around the web as ✨AverageMarcus✨ in most places and @Marcus@k8s.social on Mastodon 🐘 6+ years experience running Kubernetes in production environments.

Summary My 10 tips for working with Kubernetes #1 → #5 Anyone can start using these today #6 → #7 Good to know a little old-skool ops first #8 → #10 Good have some programming knowledge

#0 - Pay someone else to deal with it OK, this one is kinda tongue in cheek but worth mentioning. If you have dozens or hundreds of clusters on-top of other development work you’re going to be stretched thin. Getting someone else to manage things while you focus on what makes your business money can often be the right choice.

#1 - Love your terminal ★ Bash? ZSH? Fish? 🤷 - Doesn’t matter as long as you’re comfortable with it. ★ “rc” files - e.g. .bashrc, .zshrc These set configuration for each terminal session you open. ★ alias - easily create your own terminal commands ★ Look for “dotfiles” on GitHub - e.g. https://github.com/averagemarcus/dotfiles Create your own workflow of tasks you perform often. Avoid typos and ”fat fingering” by replacing long, complex commands with short aliases (bonus points for adding help text to remind you later)

#2 - Learn to love `kubectl` ★ Tip #1 in action Add alias k=’kubectl’ to your .bashrc / .zshrc / .whateverrc KIND: VERSION: k get pods -A ★ The official docs offer a single page view of all built in commands: kubernetes.io/docs/reference/generated/kube ctl/kubectl-commands ★ k explain pods.spec.containers kubectl explain is your friend! Find out what any property of any Kubernetes resource is for. ➡ Pod v1 RESOURCE: containers <[]Object> DESCRIPTION: List of containers belonging to the pod. Containers cannot currently be added or removed. There must be at least one container in a Pod. Cannot be updated. A single application container that you want to run within a pod. FIELDS: args <[]string> Arguments to the entrypoint. The docker image’s CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container’s environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. “$$(VAR_NAME)” will produce the string literal “$(VAR_NAME)”. Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. command <[]string> Entrypoint array. Not executed within a shell. The docker image’s ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) Save time by only typing `k`. Kubectl explain for digging into resources and their properties (useful when you can’t access the official docs or know exactly what you’re looking for)

#3 - Multiple kubeconfigs ★ Quick switch between different Kubernetes contexts (clusters) and between different namespaces. ★ kubectx and kubens https://github.com/ahmetb/kubectx ★ kubie https://github.com/sbstp/kubie ★ kubeswitch https://github.com/danielfoehrKn/kubeswitch kubeswitch my fave as it supports directory of kubeconfigs to make organising easier - you can see it in action here

#4 - k9s github.com/derailed/k9s Interactive terminal. Supports all resource types and actions. Lots of keybinding and similar to quickly work with a cluster. Find, view, edit, port forward, view logs, delete, etc.

#4 - OpenLens github.com/MuhammedKalkan/OpenLens Alternatively, if you prefer a more UI based interface then Lens is a great tool to manage multiple clusters.

#5 - kubectl plugins ★ Any command in your $PATH that is prefixed with kubectl- becomes a kubectl plugin $ cat kubectl-hello #!/bin/bash ★ Krew - package manager for kubectl plugins echo “Hello, Kube” github.com/kubernetes-sigs/krew ★ Install plugins with: kubectl krew install <PLUGIN NAME> $ kubectl hello Hello, Kube ★ Some of my fave plugins: ○ stern - Multi-pod/container log tailing ○ tree - Show hierarchy of resources based on ownerReferences ○ community-images - Find images still referencing the k8s.gcr.io registry. ○ gs - Giant Swarm’s plugin for working with our managed clusters Plugins can be in any language. You can easily add your own by creating Bash scripts with a `kubectl-` prefixed name. Note: autocomplete is a bit trickier here. Some plugins support it but generally expect your tabcompletion to only recommend core kubectl features.

#6 - Pod Debugging: kshell Launch a temporary pod running a bash shell for cluster debugging Tip #1 in action, again alias kshell=’kubectl run \ -it \ —image bash \ —restart Never \ —rm \ shell’ Need more tools? Replace this with alpine or ubuntu Great for more general debugging of a cluster, especially with networking issues or similar.

#6 - Pod Debugging: kshell Launch a temporary pod running a bash shell for cluster debugging
kshell If you don’t see a command prompt, try pressing enter. bash-5.1# nslookup google.com Server: 1.1.1.1 Address: 1.1.1.1:53 Non-authoritative answer: Name: google.com Address: 142.250.187.206
Great for more general debugging of a cluster, especially with networking issues or similar.

#6 - Pod Debugging: kubectl exec Debugging an existing, running pod - kubectl exec
kubectl exec my-broken-pod -it — sh /app
Note: ★ ★ ★ Needs a shell environment within the container Limited to what’s available in the container (or what you can pull in from the ‘net) Container needs to be running `kubectl exec` is great for debugging misconfigured pods that aren’t crashing and have enough OS to exec into. But… If the pod is CrashLooping you’ll get kicked out of the session when it crashes. If the pod doesn’t have a shell you can exec into (e.g. a container that only has a Golang binary) you’ll not be able to exec `kubectl debug` is great for pods that either don’t have any OS

#6 - Pod Debugging: kubectl debug Debugging a running pod - kubectl exec
kubectl exec my-broken-pod -it — sh
�� error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec…… Debugging a running pod - kubectl debug Requires Kubernetes v1.23+
kubectl debug -it —image bash my-broken-pod Defaulting debug container name to debugger-gprmk. If you don’t see a command prompt, try pressing enter. bash-5.1#
`kubectl exec` is great for debugging misconfigured pods that aren’t crashing and have enough OS to exec into. But… If the pod is CrashLooping you’ll get kicked out of the session when it crashes. If the pod doesn’t have a shell you can exec into (e.g. a container that only has a Golang binary) you’ll not be able to exec `kubectl debug` is great for pods that either don’t have any OS ��

#6 - Pod Debugging: kubectl debug Example - investigate a CrashLooping pod
kubectl run debug-demo —image=bash — exit 1 # kubectl get pods debug-demo NAME debug-demo
READY 0/1 STATUS CrashLoopBackOff RESTARTS 2 (20s ago) AGE 44s This will prevent us from `kubectl exec` into the pod
kubectl debug -it —image bash debug-demo Defaulting debug container name to debugger-5mkjj. If you don’t see a command prompt, try pressing enter. bash-5.1#
`kubectl debug` has a few different modes: - launches an “ephemeral container” within the pod you’re debugging - `kubectl debug` - creates a copy of the pod with some values replaced (e.g. the image used) - `kubectl debug –copy-to` - launch a pod in the nodes host namespace to debug the node - `kubectl debug node/my-node` This has some limitations - cannot access all filesystem of failing container, only volumes that are shared

#6 - Pod Debugging When to use what: kshell Multiple workloads experiencing network issues Workload not running as expected but not CrashLooping and isn’t a stripped down image (e.g. not Scratch / Distroless) Workload not running as expected but not CrashLooping and has an image based on Scratch / Distroless or similar Workload is CrashLooping kubectl exec kubectl debug ✅ ✅ ✅ ✅

#7 - Node Debugging: kubectl debug (again) ★ Requires Kubernetes v1.23 # kubectl debug node/ip-10-0-0-1 -it —image alpine Creating debugging pod node-debugger-ip-10-0-0-1-9wlqp with container debugger on node ip-10-0-0-1. If you don’t see a command prompt, try pressing enter. / # ls -l / total 60 … drwxr-xr-x 2 root root 4096 Aug 9 08:47 home drwxr-xr-x 19 root root 4096 Nov 4 08:48 host The host node’s root filesystem drwxr-xr-x 7 root root 4096 Aug 9 08:47 lib drwxr-xr-x 5 root root 4096 Aug 9 08:47 media … / # Why not SSH? - I prefer to use ephemeral instances with the minimal needed to run Kubernetes, no sshd, no port 22 open etc. but there are times when you just need to check what’s actually going on with the underlying host machine. `kubectl debug` launches a pod on the node which allows for direct access to the node host. Processes and filesystem is available to you. Being your own toolkit with you depending on what `--image` you specify.

#7 - Node Debugging: nsenter alternative ★ For older clusters before Kubernetes v1.23 # kubectl run h0nk —rm -it \ —image alpine —privileged \ —overrides ‘{“spec”:{“hostPID”: true}}’\ —command nsenter – \ —mount=/proc/1/ns/mnt If you don’t see a command prompt, try pressing enter. # ★ Alternatives: github.com/AverageMarcus/kube-ssh github.com/giantswarm/kubectl-enter This won’t work with Talos, for example, whereas `kubectl debug` will ★ Note: Underlying host needs a valid shell Only needed if `kubectl debug` not available for your cluster. A one-liner to exec into a privileged pod and switch to the host namespace using `nsenter` - small enough to fit in a Tweet or on a sticker! More “packaged” alternatives available with kube-ssh or kubectl-enter

Webhooks ★ Implement more advanced access control than is possible with RBAC. [Restricting cluster-admin permissions] ★ Add defaulting logic to Kubernetes resources ★ Enforce company policies such as not using latest as an image tag or ensuring all workloads have resource requests/limits specified. ★ “Hotfix” for security issues (e.g. injecting env var to prevent Log4Shell exploit). [Log4Shell Mitigation] ⚠ Be careful using webhooks as it’s easy to introduce cluster-breaking configurations! 😱 [Webhooks Talk] Tools: ★ Kyverno - Kubernetes native policy management. ★ OPA Gatekeeper - Policy management built on top of Open Policy Agent Allows for subtractive access control (take away a users ability to perform a certain action against a certain resource) - something not possible with RBAC See blog post about how we avoided a nasty bug in our CLI tool with a ValidatingWebhook. https://www.giantswarm.io/blog/restricting-cluster-admin-permissions

Kubernetes API Resources: - kubernetes/client-go - the official Golang module for interacting with the Kubernetes API - Kubernetes Provider for Terraform (actually uses the above Go module under the hood) - kubernetes-client org on GitHub has many official clients in different languages Where is this useful? ★ Building our own CLI / desktop tooling (e.g. k9s, Lens). ★ Cluster automation - resources managed by CI, CronJobs, etc. ★ Building our own operators to extend Kubernetes. Make use of one of the many client libraries available rather than interacting with the REST endpoint directly. Plenty more official clients available at https://github.com/kubernetes-client

CRDs & Operators Extend Kubernetes’ built-in API and functionality with your own Custom Resource Definitions (CRDs) and business logic (operators). Image credits: Container Solutions https://blog.container-solutions.com/kubernetes-operators-explained We’ve already seen hints of this already in this talk. Kyverno implements CRDs and has an operator that manages them. We mentioned extending the Kubernetes API with custom resources, giving us the programmatic access for our operators.

#10 - CRDs & Operators Frameworks Metacontroller References ● https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ ● https://blog.container-solutions.com/kubernetes-operators-explained ● https://operatorhub.io/ - Directory of existing operators Videos This topic is too large to cover within this talk, there are already plenty of better resources available. Kubebuilder tends to be the most popular framework and used by all of the cluster-api projects.

Recap #1 - Love your terminal #6 - Pod Debugging #2 - Learn to love kubectl #7 - Node Debugging #3 - Multiple kubeconfigs #8 - Webhooks #4 - k9s / OpenLens #9 - Kubernetes API #5 - Kubectl plugins #10 - CRDs & Controllers
Shell aliases and helpers Alias k, kubectl explain Kubeswitch Interactively work with clusters Krew. Build your own with bash. Kubectl- prefixed name Pod debugging Node debugging Validating and mutating requests to the Kubernetes API Working directly with the API to build our own logic Extending Kubernetes with our own resources and logic

Marcus Noble
@averagemarcus

1 / 36

After several years operating and building on Kubernetes clusters I have my fair share of battle scars. From network misconfigurations to soaring cloud costs to deletion of a production control plane, just to name a few.

Each time something went wrong, dealing with it became a little easier and each incident added another tool to my bag of tricks.

Over time I’ve tried to simplify and automate as much of my debugging technique as I can, through a combination of scripts, tools and code.

This talk will cover some practical tips and tricks that everyone using Kubernetes, regardless of experience level, can leverage to make interacting with clusters that little bit easier. We’ll start small, something anyone can do today, and work our way up to building our own tooling on top of Kubernetes to make every day operations just that little nicer.

Resources

The following resources were mentioned during the presentation or are useful additional information.

My dotfiles

My personal collection of shell scripts and alias commands
Kubectl commands reference

A single page view of all built in kubectl commands
kubectx & kubens

Manage multiple Kubeconfig files
kubie

Manage multiple Kubeconfig files
kubeswitch

Manage multiple Kubeconfig files
k9s

Interactive terminal-based UI
OpenLens

Interactive desktop UI
Krew

Kubectl plugins package manager
Ian Coldwater - h0nk one-liner

nsenter alternative to kubectl debug, small enough to fit in a Tweet
kube-ssh

My script to fake SSH access to Kubernetes nodes
kubectl-enter

Kubectl plugin to get ssh-like access to a node
nsenter - man page
Restricting cluster-admin permissions

My blog post on using Kyverno to hotfix a bug in our Giant Swarm CLI
The Wonders and Woes of Webhooks

Another of my talks covering Kubernetes webhooks in much more detail
Log4Shell mitigation policy for Kyverno

A Kyverno policy to mitigate the Log4Shell vulnerability cluster-wide
The Wonders and Woes of Webhooks

My confernce talk on the potential dangers of using Kubernetes webhooks.
Kyverno

Kubernetes native policy management
OPA Gatekeeper

Policy management build on top of Open Policy Agent
Go client for Kubernetes API

The official Go library for Kubernetes clients
Kubernetes Terraform provider
All official Kubernetes client libraries

Libraries for many different programming languages for working with the Kubernetes API
Kubernetes Operators Explained by Container Solutions
Kubebuilder
Operator Framework
kudo
Metacontroller
Kubernetes Operator Pattern
Operator Hub

Directory of existing operators
Writing a Kubernetes Operator from Scratch by Civo
Tutorial: Zero to Operator in 90 Minutes! by Solly Ross, Google
Smooth Operator - A Rough Guide to Kubernetes Operators by Olive Power, VMware
My blog post covering an older version of this talk

Buzz and feedback

Here’s what was said about this presentation on social media.

Thanks to everyone that came to my @dddeastmidlands talk today! 💙

Here’s the slides and links to everything I mentioned:https://t.co/CMxthSHlOV
— Marcus Noble (@Marcus_Noble_) October 7, 2023

kshell If you don’t see a command prompt, try pressing enter. bash-5.1# nslookup google.com Server: 1.1.1.1 Address: 1.1.1.1:53 Non-authoritative answer: Name: google.com Address: 142.250.187.206

kubectl exec my-broken-pod -it — sh /app

kubectl exec my-broken-pod -it — sh

kubectl debug -it —image bash my-broken-pod Defaulting debug container name to debugger-gprmk. If you don’t see a command prompt, try pressing enter. bash-5.1#

kubectl run debug-demo —image=bash — exit 1 # kubectl get pods debug-demo NAME debug-demo

kubectl debug -it —image bash debug-demo Defaulting debug container name to debugger-5mkjj. If you don’t see a command prompt, try pressing enter. bash-5.1#

Link for this presentation:

HTML code for embedding:

Share on social media: