Pause Container ● Every Pod includes an empty pause container which bootstraps the Pod with the cgroups, reservations and namespaces before the defined containers are created ● This container can be thought of as a “parent container” for all the containers within your Pod and will remain even if workload containers crash, ensuring namespaces and networking remain available ● The pause container is always present but not visible via the Kubernetes API ● Can be seen if you query directly on the node, e.g. with containerd: ~ # ctr -n k8s.io containers list | grep pause 03bfc9fa4bd0aebb0a9f84b1aad680f4b7 gsoci.azurecr.io/giantswarm/pause:3.9 io.containerd.runc.v2 05df0356a344c23d02afbff797742c67bd gsoci.azurecr.io/giantswarm/pause:3.9 io.containerd.runc.v2 066e0c8ee2962f276c4b7bb7d505e63f5b gsoci.azurecr.io/giantswarm/pause:3.9 io.containerd.runc.v2 0a6685e4d54e94c4acc36dbbb1a2b356de gsoci.azurecr.io/giantswarm/pause:3.9 io.containerd.runc.v2 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

RuntimeClass ● Allows for multiple runtimes in a single cluster ● If unset, uses default Container Runtime Interface (CRI) configured on the node ● If set, must point to a RuntimeClass resource name and have the CRI handler configured up on the node ● The scheduling property of the RuntimeClass ensures Pods are scheduled onto nodes with that runtime available (based on label selectors) apiVersion: v1 kind: Pod metadata: name: “tiny-pod” spec: runtimeClassName: “crio-runtime” containers: - name: “demo” image: “nginx:latest” —apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: “crio-runtime” scheduling: nodeSelector: runtime: “crio” handler: “crio” 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

RuntimeClass ● Allows for multiple runtimes in a single cluster ● If unset, uses default Container Runtime Interface (CRI) configured on the node ● If set, must point to a RuntimeClass resource name and have the CRI handler configured up on the node ● The scheduling property of the RuntimeClass ensures Pods are scheduled onto nodes with that runtime available (based on label selectors) ● Can be used for WASM (web assembly) runtimes, not just containers apiVersion: v1 kind: Pod metadata: name: “wasm-pod” spec: runtimeClassName: “wasmedge” containers: - name: “demo” image: “my-wasm-demo:latest” —apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: “wasmedge” scheduling: nodeSelector: runtime: “wasmedge” handler: “wasmedge” 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Static Pods ● Managed directly by the Kubelet, not the API server ● Defined as static manifests either: ○ on disk of the node in the directory defined by —pod-manifest-path ○ or referenced from an URL using the —manifest-url flag ● The Kubelet automatically tries to create a “mirror Pod” on the API for each static Pod so that they are visible when querying the API server but they cannot be modified via the API ● Pod names get the node name as a suffix (e.g. kube-scheduler-control-plane-1) ● Cannot refer to other resources (e.g. ConfigMaps) ● The Kubelet watches the static directory and reconciles when files are changed/added/removed File stored on the host node disk apiVersion: v1 kind: Pod metadata: name: kube-scheduler namespace: kube-system spec: containers: - name: kube-scheduler image: kube-scheduler:v1.32.0 command: - kube-scheduler 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Lifecycle Hooks ● Guaranteed to trigger at least once but may be called multiple times. ● postStart ● Runs immediately after container is created but no guarantee that it will execute before the container’s ENTRYPOINT. ○ The container isn’t marked as “running” until this completes. preStop ○ ○ ● Runs immediately before the container is terminated. Hook mechanisms available: ○ ○ Example based on the Cilium chart provided by Bitnami apiVersion: v1 kind: Pod metadata: name: cilium-agent spec: containers: - name: cilium-agent image: “cilium:latest” lifecycle: postStart: exec: command: - /bin/bash - -ec - | if [[ “$(iptables-save | grep -E -c ‘AWS-SNAT-CHAIN)” iptables-save | grep -E -v ‘AWS-SNAT-CHAIN’ | ipta fi preStop: exec: command: - /opt/bitnami/scripts/cilium/uninstall-cni-plugin.sh - /host exec - perform command in container httpGet - perform an HTTP GET request to the container 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Conditions Kubelet manages the following Pod Conditions: ● ● ● ● ● PodScheduled - the Pod has been scheduled to a node PodReadyToStartContainers - (beta feature) the Pod sandbox has been created and networking configured ContainersReady - all containers in the Pod are ready Initialized - all the initContainers have completed Ready - all containers ready and probes successfully passing apiVersion: v1 kind: Pod metadata: name: “demo-pod” spec: … status: conditions: - type: Ready status: “False” lastProbeTime: null - type: PodScheduled status: “True” lastProbeTime: null Each status condition may also contain… ● ● A 🤖machine readable reason property and A 🧑human readable message property …that can be used for debugging. 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Readiness Gates When container status and probes aren’t enough to determine is a Pod really is ready then there is readinessGates! These must me handled by some external application that patches the status of the Pod once the readiness gate condition is met. Example usage: AWS Load Balancer supports readiness gates to indicate a pod is registered to the ALB/NLB. apiVersion: v1 kind: Pod metadata: name: “aws-alb-example” spec: readinessGates: - conditionType: “target-health.elbv2.k8s.aws/k8s-readines-perf1000-7848e5026b” status: conditions: - type: “target-health.elbv2.k8s.aws/k8s-readines-perf1000-7848e5026b” status: “False” message: “Initial health checks in progress” reason: “Elb.InitialHealthChecking” 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Resource Requests & Limits ● Requests - min resources free on node to be scheduled ● Limits - enforced amount of resources a container has ○ CPU - enforced by CPU throttling ○ Memory - enforced by kernel out of memory (OOM) terminations ● Custom resource types can be managed by 3rd party controllers (e.g. nvidia.com/gpu) ○ Requests & limits must be the same ○ You cannot specify requests without limits ● Pod limit and requests are calculated from the sum of all the containers ○ v1.32 introduces a new alpha (disabled by default) feature that supports pod-level resource specification apiVersion: v1 kind: Pod metadata: name: “demo-pod” spec: resources: requests: memory: “100Mi” limits: memory: “200Mi” containers: - name: demo image: nginx 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Node Assignment ● topologySpreadConstraints - more control over the spread of Pods across a cluster when scaling replicas ○ ○ ○ ○ ○ skew = number of pods in topology - min pods in any topology topologyKey - the label on nodes to use as the groupings (usually failure domains) whenUnsatisfiable - Either DoNotSchedule or ScheduleAnyway No guarantee the constraints remain satisfied when Pods are removed (e.g. scaling down) Combined with other node assignment strategies (e.g. affinity) apiVersion: v1 kind: Pod metadata: name: “demo-pod” labels: app: nginx spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: nginx containers: - name: demo image: nginx 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Priority & Preemption ● Pods can be given a priority to indicate their importance compared to other Pods. If a Pod is unable to be scheduled and has a higher priority than already scheduled Pods the scheduler will evict (preempt) the lower priority to make room ● PriorityClass resource is used to define possible priorities in a cluster ● PodDisruptionBudget are handled on a best-effort basis and not guaranteed to be honoured ● You can avoid preempting lower priority Pods by setting preemptionPolicy: Never on the PriorityClass ○ This effects the scheduler queue but doesn’t cause pods to be evicted apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority value: 1000000 globalDefault: false description: These pods are important —apiVersion: v1 kind: Pod metadata: name: demo spec: priorityClassName: high-priority containers: - name: demo image: nginx 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

DNS ● Usually, depending on the DNS mechanism used in the cluster (e.g. CoreDNS), each Pod also gets an A record ○ ● ● E.g. 172-17-0-3.default.pod.cluster.local A Pods hostname is set to its metadata.name by default but can be overridden with the spec.hostname property and an additional subdomain set with spec.subdomain ○ E.g. my-demo.example.default.svc.cluster.local ○ (This doesn’t mean other Pods can resolve that hostname) Add extra entries to the Pods /etc/hosts file with spec.hostAliases apiVersion: v1 kind: Pod metadata: name: demo spec: hostname: my-demo subdomain: example setHostnameAsFQDN: true hostAliases: - ip: “127.0.0.1” hostnames: - “demo.local” containers: - name: demo image: nginx 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

DNS Config ● More control over the DNS settings used within a Pod ● nameservers - a list of IPs to use as the DNS servers (max 3) ● searches - list of DNS search domains to use for hostname lookup, merged into the base search domains generated (max 32) ● options - a list of name/value pairs to define extra DNS configuration options

cat /etc/resolv.conf nameserver 192.0.2.1 search ns1.svc.cluster.example my.dns.custom options ndots:2 edns0

apiVersion: v1 kind: Pod metadata: name: demo spec: containers: - name: busybox Only use the below config image: nginx dnsPolicy: “None” dnsConfig: nameservers: - 192.0.2.1 searches: - ns1.svc.cluster.example - my.dns.custom options: - name: ndots value: “2” - name: edns0 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social