Pod Deep Dive The Interesting Bits KCD Czech & Slovak June 5th 2025 ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
A presentation at KCD Czech & Slovak 2025 in June 2025 in Bratislava, Slovakia by Marcus Noble
Pod Deep Dive The Interesting Bits KCD Czech & Slovak June 5th 2025 ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Hi ๐, Iโm Marcus Noble! Iโm a platform engineer at Giant Swarm working on release engineering, CI/CD and general Kubernetes development. I run a monthly newsletter - CloudNative.Now 7+ years experience running Kubernetes in production environments. ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
So what is a โPodโ? ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Pods are the smallest deployable units of computing that you can create and manage in Kubernetes. https://kubernetes.io/docs/concepts/workloads/pods/ ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
So what is a โPodโ? โ โ โ โ โ โ Our containers / WASM Smallest deployable unit of computing A wrapper around one or more containers (or WASM functions) Managed by the Kubernetes scheduler and assigned to nodes at runtime Workloads within a pod all share the same runtime context (Linux namespaces, cgroups, network, etc) Designed to be relatively ephemeral and disposable Mostly immutable (only changes to image, activeDeadlineSeconds and additions to tolerations are allowed) โ v1.33 graduated in-place Pod resize to beta that allows changes to resources also ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
So what is a โPodโ? But really, itโs much more than that! Too much to cover in fact! So letโs talk about the interesting, weird or surprising bits! ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Containers ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Sidecar Containers โ Enabled by default from v1.29 (feature gate SidecarContainers) โ โDisguisedโ as initContainers ๐คท โ Launched when Pod scheduled, continues running until main application containers have fully stopped, then kubelet terminates all sidecars โ Supports readinessProbe (unlike normal initContainers) and used to determine the ready state of the Pod โ Termination handled more harshly than app containers - SIGTERM followed by SIGINT before graceful exit likely apiVersion: v1 kind: Pod metadata: name: โtiny-podโ spec: initContainers: - name: logshipper image: alpine Only allowed value restartPolicy: Always On the container, not the Pod ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Ephemeral Containers โ Designed for debugging โ Must be added via a special ephemeralcontainers handler, not via the Pod spec (e.g. kubectl debug) โ Not supported on static pods โ Can target specific container process namespaces with optional targetContainerName property โ No support for ports, probes, resources or lifecycle on the container spec kubectl debug tiny-pod -it \ โimage=alpine \ โtarget=nginx apiVersion: v1 kind: Pod metadata: name: โtiny-podโ Read only spec: ephemeralContainers: - name: debugger-67t9x image: alpine targetContainerName: nginx ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Pause Container โ Every Pod includes an empty pause container which bootstraps the Pod with the cgroups, reservations and namespaces before the defined containers are created โ This container can be thought of as a โparent containerโ for all the containers within your Pod and will remain even if workload containers crash, ensuring namespaces and networking remain available โ The pause container is always present but not visible via the Kubernetes API โ Can be seen if you query directly on the node, e.g. with containerd: ~ # ctr -n k8s.io containers list | grep pause 03bfc9fa4bd0aebb0a9f84b1aad680f4b7 gsoci.azurecr.io/giantswarm/pause:3.9 io.containerd.runc.v2 05df0356a344c23d02afbff797742c67bd gsoci.azurecr.io/giantswarm/pause:3.9 io.containerd.runc.v2 066e0c8ee2962f276c4b7bb7d505e63f5b gsoci.azurecr.io/giantswarm/pause:3.9 io.containerd.runc.v2 0a6685e4d54e94c4acc36dbbb1a2b356de gsoci.azurecr.io/giantswarm/pause:3.9 io.containerd.runc.v2 ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Images ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Image Pull Policy โ IfNotPresent - Fetches the image if not already on the node (default if you use a tag/sha) โ Always - Will always fetch the image (default if you use the tag as โlatestโ or omit the tag) โ โ The cache mechanism compares the image layers from the registry and only pulls those missing Never - Will not attempt to fetch the image, it must be loaded onto the node by some other means apiVersion: v1 kind: Pod metadata: name: โtiny-podโ spec: containers: - name: โnginxโ image: โnginx:v1.2.3โ imagePullPolicy: โAlwaysโ ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Image Tags - SHA โ Recommended best practice โ SHA-based image tag ensure exactly the same image is used each time, even if tag it overwritten โ If SHA is used, the tag is completely ignored and may no longer match the SHA! โ โ Be careful with automated dependency updaters - make sure the sha is also updated! apiVersion: v1 kind: Pod metadata: name: โtiny-podโ spec: containers: Meaningless / Ignored - name: โnginxโ image: โnginx:1.25.1@sha256:9d6b58feebd2dbโฆ2072c9496โ imagePullPolicy: โAlwaysโ ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
RuntimeClass ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
RuntimeClass โ Allows for multiple runtimes in a single cluster โ If unset, uses default Container Runtime Interface (CRI) configured on the node โ If set, must point to a RuntimeClass resource name and have the CRI handler configured up on the node โ The scheduling property of the RuntimeClass ensures Pods are scheduled onto nodes with that runtime available (based on label selectors) apiVersion: v1 kind: Pod metadata: name: โtiny-podโ spec: runtimeClassName: โcrio-runtimeโ containers: - name: โdemoโ image: โnginx:latestโ โapiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: โcrio-runtimeโ scheduling: nodeSelector: runtime: โcrioโ handler: โcrioโ ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
RuntimeClass โ Allows for multiple runtimes in a single cluster โ If unset, uses default Container Runtime Interface (CRI) configured on the node โ If set, must point to a RuntimeClass resource name and have the CRI handler configured up on the node โ The scheduling property of the RuntimeClass ensures Pods are scheduled onto nodes with that runtime available (based on label selectors) โ Can be used for WASM (web assembly) runtimes, not just containers apiVersion: v1 kind: Pod metadata: name: โwasm-podโ spec: runtimeClassName: โwasmedgeโ containers: - name: โdemoโ image: โmy-wasm-demo:latestโ โapiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: โwasmedgeโ scheduling: nodeSelector: runtime: โwasmedgeโ handler: โwasmedgeโ ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Static Pods ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Static Pods โ Managed directly by the Kubelet, not the API server โ Defined as static manifests either: โ on disk of the node in the directory defined by โpod-manifest-path โ or referenced from an URL using the โmanifest-url flag โ The Kubelet automatically tries to create a โmirror Podโ on the API for each static Pod so that they are visible when querying the API server but they cannot be modified via the API โ Pod names get the node name as a suffix (e.g. kube-scheduler-control-plane-1) โ Cannot refer to other resources (e.g. ConfigMaps) โ The Kubelet watches the static directory and reconciles when files are changed/added/removed File stored on the host node disk apiVersion: v1 kind: Pod metadata: name: kube-scheduler namespace: kube-system spec: containers: - name: kube-scheduler image: kube-scheduler:v1.32.0 command: - kube-scheduler ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Lifecycle Hooks ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Lifecycle Hooks โ Guaranteed to trigger at least once but may be called multiple times. โ postStart โ Runs immediately after container is created but no guarantee that it will execute before the containerโs ENTRYPOINT. โ The container isnโt marked as โrunningโ until this completes. preStop โ โ โ Runs immediately before the container is terminated. Hook mechanisms available: โ โ Example based on the Cilium chart provided by Bitnami apiVersion: v1 kind: Pod metadata: name: cilium-agent spec: containers: - name: cilium-agent image: โcilium:latestโ lifecycle: postStart: exec: command: - /bin/bash - -ec - | if [[ โ$(iptables-save | grep -E -c โAWS-SNAT-CHAIN)โ iptables-save | grep -E -v โAWS-SNAT-CHAINโ | ipta fi preStop: exec: command: - /opt/bitnami/scripts/cilium/uninstall-cni-plugin.sh - /host exec - perform command in container httpGet - perform an HTTP GET request to the container ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Conditions & Readiness Gates ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Conditions Kubelet manages the following Pod Conditions: โ โ โ โ โ PodScheduled - the Pod has been scheduled to a node PodReadyToStartContainers - (beta feature) the Pod sandbox has been created and networking configured ContainersReady - all containers in the Pod are ready Initialized - all the initContainers have completed Ready - all containers ready and probes successfully passing apiVersion: v1 kind: Pod metadata: name: โdemo-podโ spec: โฆ status: conditions: - type: Ready status: โFalseโ lastProbeTime: null - type: PodScheduled status: โTrueโ lastProbeTime: null Each status condition may also containโฆ โ โ A ๐คmachine readable reason property and A ๐งhuman readable message property โฆthat can be used for debugging. ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Readiness Gates When container status and probes arenโt enough to determine is a Pod really is ready then there is readinessGates! These must me handled by some external application that patches the status of the Pod once the readiness gate condition is met. Example usage: AWS Load Balancer supports readiness gates to indicate a pod is registered to the ALB/NLB. apiVersion: v1 kind: Pod metadata: name: โaws-alb-exampleโ spec: readinessGates: - conditionType: โtarget-health.elbv2.k8s.aws/k8s-readines-perf1000-7848e5026bโ status: conditions: - type: โtarget-health.elbv2.k8s.aws/k8s-readines-perf1000-7848e5026bโ status: โFalseโ message: โInitial health checks in progressโ reason: โElb.InitialHealthCheckingโ ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Config ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Config Environment Variables ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Environment Variables โ Hardcoded & Dynamic, leveraging other environment variables with the $(ENV) syntax apiVersion: v1 kind: Pod metadata: name: โdemo-podโ spec: containers: - name: demo image: nginx env: - name: NAME value: โWorldโ - name: GREETING value: โHello, $(NAME)โ ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Environment Variables โ Hardcoded & Dynamic, leveraging other environment variables with the $(ENV) syntax โ The Downward API allows exposing properties from the Pod fields as env vars. Not all fields are valid but you can use fields from the Podโs metadata, spec, limits and status. apiVersion: v1 kind: Pod metadata: name: โdemo-podโ spec: containers: - name: demo image: nginx env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP - name: CONTAINER_MEM_LIMIT valueFrom: resourceFieldRef: containerName: demo resource: limits.memory ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Config Volumes ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Volumes โ ConfigMaps vs. Secrets - name vs. secretName apiVersion: v1 kind: Pod metadata: name: โdemo-podโ spec: containers: - name: demo image: nginx volumes: - name: config-vol configMap: name: sensitive-html - name: secret-vol secret: secretName: demo-html ๏ฟฝ๏ฟฝ ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Volumes โ ConfigMaps vs. Secrets - name vs. secretName โ Downward API apiVersion: v1 kind: Pod metadata: name: โdemo-podโ spec: containers: - name: demo image: nginx volumeMounts: - name: podinfo mountPath: /etc/podinfo volumes: - name: podinfo downwardAPI: Becomes the filename items: - path: โlabelsโ fieldRef: fieldPath: metadata.labels - path: โannotationsโ fieldRef: fieldPath: metadata.annotat ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Volumes โ ConfigMaps vs. Secrets - name vs. secretName โ Downward API โ EmptyDir apiVersion: v1 kind: Pod metadata: name: โdemo-podโ spec: containers: - name: demo image: nginx volumes: - name: cache-volume emptyDir: medium: Memory sizeLimit: 500Mi Recommended to avoid filling host node disk ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Volumes โ ConfigMaps vs. Secrets - name vs. secretName โ Downward API โ EmptyDir โ Projected apiVersion: v1 kind: Pod metadata: name: โdemo-podโ spec: containers: - name: demo image: nginx volumeMounts: - name: web-content mountPath: /usr/share/nginx/html volumes: - name: web-content projected: sources: - configMap: name: web-index items: - key: index.html path: index.html - configMap: name: error-pages Entire contents of ConfigMap data ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Volumes โ โ โ โ โ apiVersion: v1 kind: Pod metadata: ConfigMaps vs. Secrets - name vs. secretName name: โdemo-podโ spec: Downward API containers: - name: demo EmptyDir image: nginx Projected volumeMounts: - name: oci-content Image (KEP #4639) mountPath: /usr/share/nginx/html readOnly: true โ Alpha in v1.31, Beta in v1.33, disabled by default volumes: - name: oci-content โ Allows mounting an OCI image as a volume image: reference: quay.io/crio/artifact:v1 โ Pull secrets handled the same as container images pullPolicy: IfNotPresent โ Container runtime needs to support it (CRI-O and Containerd have initial support available) ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Scheduling ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Scheduling Resource Allocation ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Very brief overview! Resource Requests & Limits โ Requests - min resources free on node to be scheduled โ Limits - enforced amount of resources a container has โ CPU - enforced by CPU throttling โ Memory - enforced by kernel out of memory (OOM) terminations apiVersion: v1 kind: Pod metadata: name: โdemo-podโ spec: containers: - name: demo image: nginx resources: requests: memory: โ64Miโ cpu: โ250mโ limits: memory: โ128Miโ cpu: โ500mโ ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Resource Requests & Limits โ Requests - min resources free on node to be scheduled โ Limits - enforced amount of resources a container has โ CPU - enforced by CPU throttling โ Memory - enforced by kernel out of memory (OOM) terminations โ Custom resource types can be managed by 3rd party controllers (e.g. nvidia.com/gpu) โ Requests & limits must be the same โ You cannot specify requests without limits For GPU resources apiVersion: v1 kind: Pod metadata: name: โdemo-podโ spec: containers: - name: demo image: nginx resources: requests: nvidia.com/gpu: 1 limits: nvidia.com/gpu: 1 ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Resource Requests & Limits โ Requests - min resources free on node to be scheduled โ Limits - enforced amount of resources a container has โ CPU - enforced by CPU throttling โ Memory - enforced by kernel out of memory (OOM) terminations โ Custom resource types can be managed by 3rd party controllers (e.g. nvidia.com/gpu) โ Requests & limits must be the same โ You cannot specify requests without limits โ Pod limit and requests are calculated from the sum of all the containers โ v1.32 introduces a new alpha (disabled by default) feature that supports pod-level resource specification apiVersion: v1 kind: Pod metadata: name: โdemo-podโ spec: resources: requests: memory: โ100Miโ limits: memory: โ200Miโ containers: - name: demo image: nginx ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Scheduling Node Assignment ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Node Assignment โ topologySpreadConstraints - more control over the spread of Pods across a cluster when scaling replicas โ โ โ โ โ skew = number of pods in topology - min pods in any topology topologyKey - the label on nodes to use as the groupings (usually failure domains) whenUnsatisfiable - Either DoNotSchedule or ScheduleAnyway No guarantee the constraints remain satisfied when Pods are removed (e.g. scaling down) Combined with other node assignment strategies (e.g. affinity) apiVersion: v1 kind: Pod metadata: name: โdemo-podโ labels: app: nginx spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: nginx containers: - name: demo image: nginx ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Scheduling Scheduler Logic ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Priority & Preemption โ Pods can be given a priority to indicate their importance compared to other Pods. If a Pod is unable to be scheduled and has a higher priority than already scheduled Pods the scheduler will evict (preempt) the lower priority to make room โ PriorityClass resource is used to define possible priorities in a cluster โ PodDisruptionBudget are handled on a best-effort basis and not guaranteed to be honoured โ You can avoid preempting lower priority Pods by setting preemptionPolicy: Never on the PriorityClass โ This effects the scheduler queue but doesnโt cause pods to be evicted apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority value: 1000000 globalDefault: false description: These pods are important โapiVersion: v1 kind: Pod metadata: name: demo spec: priorityClassName: high-priority containers: - name: demo image: nginx ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Multiple / Alternative Schedulers โ schedulerName - indicates which scheduler a Pod should be managed by โ In not set, or set to default-scheduler then the built-in Kubernetes scheduler is used apiVersion: v1 kind: Pod metadata: name: demo spec: schedulerName: custom-scheduler containers: - name: demo image: nginx ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Networking ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
DNS โ Usually, depending on the DNS mechanism used in the cluster (e.g. CoreDNS), each Pod also gets an A record โ โ โ E.g. 172-17-0-3.default.pod.cluster.local A Pods hostname is set to its metadata.name by default but can be overridden with the spec.hostname property and an additional subdomain set with spec.subdomain โ E.g. my-demo.example.default.svc.cluster.local โ (This doesnโt mean other Pods can resolve that hostname) Add extra entries to the Pods /etc/hosts file with spec.hostAliases apiVersion: v1 kind: Pod metadata: name: demo spec: hostname: my-demo subdomain: example setHostnameAsFQDN: true hostAliases: - ip: โ127.0.0.1โ hostnames: - โdemo.localโ containers: - name: demo image: nginx ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
DNS Policy โ Define how the Pods DNS configuration defaults should be specified: Actually the default value ๐คท โ Default - inherits the nodes DNS resolution config โ ClusterFirst - matches against in-cluster resources first before sending forwarding to an upstream nameserver โ ClusterFirstWithHostNet - should be used when using host network otherwise the Pod will fallback to Default โ None - ignores all DNS config from cluster and expects all to be set via dnsConfig apiVersion: v1 kind: Pod metadata: name: demo spec: dnsPolicy: ClusterFirst containers: - name: demo image: nginx ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
DNS Config โ More control over the DNS settings used within a Pod โ nameservers - a list of IPs to use as the DNS servers (max 3) โ searches - list of DNS search domains to use for hostname lookup, merged into the base search domains generated (max 32) โ options - a list of name/value pairs to define extra DNS configuration options
apiVersion: v1 kind: Pod metadata: name: demo spec: containers: - name: busybox Only use the below config image: nginx dnsPolicy: โNoneโ dnsConfig: nameservers: - 192.0.2.1 searches: - ns1.svc.cluster.example - my.dns.custom options: - name: ndots value: โ2โ - name: edns0 ๐ @Marcus@k8s.social | ๐ MarcusNoble.com | ๐ฆ @averagemarcus.bsky.social
Wrap-up Slides and resources available at: https://go-get.link/kcdczechslovak Thoughts, comments and feedback: MarcusNoble.com k8s.social/@Marcus @averagemarcus.bsky.social Thank you