From Fragile to Resilient: ValidatingAdmissionPolicies Strengthen Kubernetes

A presentation at Kubernetes Community Days UK in October 2024 in London, UK by Marcus Noble

Hi 👋, I’m Marcus Noble! I’m a platform engineer at working on release engineering, CI/CD infrastructure and provider-independent Kubernetes development. 6+ years experience running Kubernetes in production environments. Fediverse at: @Marcus@k8s.social Everywhere else: MarcusNoble.com

Dynamic Admission Control ValidatingAdmission MutatingAdmission Webhook Webhook

Purpose / Use Cases Defaulting Policy Enforcement ● Injecting imagePullSecrets dynamically when pods are created ● Injecting sidecars into pods ● Injecting proxy environment variables into pods ● Prevent using latest image tag ● Require all pods to have resource limits set ● Block the use of deprecated Kubernetes APIs (e.g. batch/v1beta1) ● Require a PodDisruptionBudget ● Enforce a standard set of labels / annotations on all resources ● Replace all image registries with an in-house container image proxy / cache ● Block nodes joining the cluster with known CVEs based on the kernel version (e.g. CVE-2022-0185) ● Inject Log4Shell mitigation env var into all pods (CVE-2021-44228) ● Block binding to the cluster-admin role Best Practices Problem Mitigation

Webhooks in Kubernetes are ✨ POWERFUL ✨ But with that power comes Taken from my talk about this at K UK last year 😱 RISK 😱

Wouldn’t it be great if we had a safer alternative? Yes! Yes, it would! 6

Introducing ValidatingAdmissionPolicies Status: Alpha in v1.26, Beta in v1.28, GA in v1.30 Introduced in: KEP-3488 A declarative, in-process alternative to validating admission webhooks.

Introducing ValidatingAdmissionPolicies Status: Alpha in v1.26, Beta in v1.28, GA in v1.30 Introduced in: KEP-3488 api-server A declarative, in-process alternative to validating admission webhooks. Kubernetes manifests

Brief introduction to CEL ● Uses a similar syntax to the expressions in C-based languages, e.g. self.minReplicas <= self.replicas && self.replicas <= self.maxReplicas ● Designed to be embedded into other applications with a focus on “one-liners” of code. ● Small number of built in functions (e.g. split, has) ● Functions expanded through custom libraries (Kubernetes includes several of these) References: ● ● Already used by Kyverno, Tekton, etc. ● Has a concept of “cost” per operation. https://kubernetes.io/docs/reference/usingapi/cel/ ● https://github.com/google/cel-go ● https://playcel.undistro.io/ (CEL playground) This can be calculated ahead of running the expression and if needed, block its execution if too expensive.

Example Blocking the use of the ‘latest’ image tag prevent-latest-vap.yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: “prevent-latest-image-tag” spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”, “UPDATE”] resources: [“deployments”, “daemonsets”, “statefulsets”] validations: - message: “The use of the ‘latest’ tag is not allowed” expression: | object.spec.template.spec.containers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) && ( !has(object.spec.template.spec.initContainers) || object.spec.template.spec.initContainers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) )

Example Blocking the use of the ‘latest’ image tag prevent-latest-binding.yaml Our policy does nothing until we bind it to some conditions. apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: “prevent-latest-image-tag” spec: policyName: “prevent-latest-image-tag” validationActions: [Deny] matchResources: {}

Example Blocking the use of the ‘latest’ image tag prevent-latest-binding.yaml The name of the policy we have just created. apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: “prevent-latest-image-tag” spec: policyName: “prevent-latest-image-tag” validationActions: [Deny] matchResources: {}

Example Blocking the use of the ‘latest’ image tag prevent-latest-binding.yaml The action to take when a policy isn’t met. Available options are: Deny, Warn & Audit apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: “prevent-latest-image-tag” spec: policyName: “prevent-latest-image-tag” validationActions: [Deny] Note: this is an array as you matchResources: {} can specify both warn and audit together

Example Blocking the use of the ‘latest’ image tag prevent-latest-binding.yaml We’re not defining any filtering so this policy applies cluster-wide. We could limit our policy to specific namespaces or labels, for example. apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: “prevent-latest-image-tag” spec: policyName: “prevent-latest-image-tag” validationActions: [Deny] matchResources: {}

Example Blocking the use of the ‘latest’ image tag deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-blocked labels: app: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:latest
kubectl apply -f deployment.yaml The deployments “nginx-blocked” is invalid: : ValidatingAdmissionPolicy ‘prevent-latest-image-tag’ with binding ‘prevent-latest-image-tag’ denied request: The use of the ‘latest’ tag is not allowed
Our friendly message ⛔ Blocked! ⛔

More advanced features ● More context values - Along with object you also have oldObject & request you can use in your expressions matchConditions: - name: “exclude-kubelet-requests” expression: “!(“system:nodes” in request.userInfo.groups)”

More advanced features ● More context values - Along with object you also have oldObject & request you can use in your expressions ● Parameters - make your policies configurable by allowing parameter resources to be applied by the binding policy.yaml apiVersion: admissionregistration.k8s… kind: ValidatingAdmissionPolicy metadata: name: “param-example” spec: paramKind: apiVersion: rules.example.com/v1 kind: ReplicaLimit validations: - expression: object.spec.replicas <= params.maxReplicas policy-binding.yaml apiVersion: admissionregistration.k8s… kind: ValidatingAdmissionPolicyBinding metadata: name: “param-example” spec: policyName: “param-example” paramRef: name: “my-parameters” namespace: “default” params.yaml apiVersion: rules.example.com/v1 kind: ReplicaLimit metadata: name: “my-parameters” namespace: “default” maxReplicas: 3

So that’s validation covered, what about mutating? Well… 34

KEP-3962 - MutatingAdmissionPolicies Status: Currently targeting v1.32 for alpha Work in progress implementation in #127134 ● Follows the same idea as ValidatingAdmissionPolicies. ● Introduces MutatingAdmissionPolicy and MutatingAdmissionPolicyBinding ● Supports two patch strategies: ApplyConfiguration and JSONPatch ● Phase 1 will only support adding and updating of values (unsetting values planned for phase 2 but will need to get creative as CEL doesn’t natively support such an operation) ● Not yet finalized or available for testing (unless you want to build from source)

Example Injecting proxy values as env vars proxy-vars-map.yaml apiVersion: admissionregistration.k8s.io/v1alpha1 kind: MutatingAdmissionPolicy metadata: name: “proxy-values” spec: matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”] resources: [“pods”] mutations: - patchType: “ApplyConfiguration” mutation: > Object{ spec: Object.spec{ containers: object.spec.containers.map(c, Object.spec.containers.item{ name: c.name, env: [ Object.spec.containers.env{ name: “HTTP_PROXY”, value: “http://proxy.proxy.svc:3128” } + c.env ] } ) } }

Example Injecting proxy values as env vars Same as ValidatingAdmissionPolicies proxy-vars-map.yaml apiVersion: admissionregistration.k8s.io/v1alpha1 kind: MutatingAdmissionPolicy metadata: name: “proxy-values” spec: matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”] resources: [“pods”] mutations: - patchType: “ApplyConfiguration” mutation: > Object{ spec: Object.spec{ containers: object.spec.containers.map(c, Object.spec.containers.item{ name: c.name, env: [ Object.spec.containers.env{ name: “HTTP_PROXY”, value: “http://proxy.proxy.svc:3128” } + c.env ] } ) } }

Example Injecting proxy values as env vars Replace validations with mutations proxy-vars-map.yaml apiVersion: admissionregistration.k8s.io/v1alpha1 kind: MutatingAdmissionPolicy metadata: name: “proxy-values” spec: matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”] resources: [“pods”] mutations: - patchType: “ApplyConfiguration” mutation: > Object{ spec: Object.spec{ containers: object.spec.containers.map(c, Object.spec.containers.item{ name: c.name, env: [ Object.spec.containers.env{ name: “HTTP_PROXY”, value: “http://proxy.proxy.svc:3128” } + c.env ] } ) } }

Example Injecting proxy values as env vars Indicates the patch strategy Possible values: ● ApplyConfiguration ● JSONPatch proxy-vars-map.yaml apiVersion: admissionregistration.k8s.io/v1alpha1 kind: MutatingAdmissionPolicy metadata: name: “proxy-values” spec: matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”] resources: [“pods”] mutations: - patchType: “ApplyConfiguration” mutation: > Object{ spec: Object.spec{ containers: object.spec.containers.map(c, Object.spec.containers.item{ name: c.name, env: [ Object.spec.containers.env{ name: “HTTP_PROXY”, value: “http://proxy.proxy.svc:3128” } + c.env ] } ) } }

Example Injecting proxy values as env vars Introduction of named types The Object refers to the type of the incoming resource (Pod in this example). This may change in later releases to something like v1.Pod.spec for example. This has memory consumption implications. proxy-vars-map.yaml apiVersion: admissionregistration.k8s.io/v1alpha1 kind: MutatingAdmissionPolicy metadata: name: “proxy-values” spec: matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”] resources: [“pods”] mutations: - patchType: “ApplyConfiguration” mutation: > Object{ spec: Object.spec{ containers: object.spec.containers.map(c, Object.spec.containers.item{ name: c.name, env: [ Object.spec.containers.env{ name: “HTTP_PROXY”, value: “http://proxy.proxy.svc:3128” } + c.env ] } ) } }

Example Injecting proxy values as env vars Merge our proxy vars with any existing vars in all containers. Note: We’re missing initContainers and ephermeralContainers here due to limited space. proxy-vars-map.yaml apiVersion: admissionregistration.k8s.io/v1alpha1 kind: MutatingAdmissionPolicy metadata: name: “proxy-values” spec: matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”] resources: [“pods”] mutations: - patchType: “ApplyConfiguration” mutation: > Object{ spec: Object.spec{ containers: object.spec.containers.map(c, Object.spec.containers.item{ I’m not 100% name: c.name, env: [ sure about Object.spec.containers.env{ this type name: “HTTP_PROXY”, value: “http://proxy.proxy.svc:3128” } + c.env ] } ) } }

Example Injecting proxy values as env vars proxy-vars-binding.yaml apiVersion: admissionregistration.k8s.io/v1beta1 kind: MutatingAdmissionPolicyBinding metadata: name: “proxy-values” spec: policyName: “proxy-values” matchResources: {} Nothing too surprising about our binding resource. All properties are the same as Validating except for the lack of a validationActions property.

All these are speculative / wishful thinking right now ● Generative policies ● Resource lookup ● Policy exceptions ● More abstractions (e.g. Kyverno already working on this with VAP)

Summary ● Time to start replacing those risky webhooks with in-process policies. ● ValidatingAdmissionPolicies generally available for use from v1.30 for safe validation logic. ● Keep an eye on KEP-3962 for the status of MutatingAdmissionPolicies. Currently targeting v1.32 for alpha release. ● More abstractions, generative policies and api lookups hopefully coming in the future.

Wrap-up Slides and resources available at: https://go-get.link/kcduk24 Thoughts, comments and feedback: feedback@marcusnoble.co.uk https://k8s.social/@Marcus Thank you

Marcus Noble
@averagemarcus

1 / 46

In the world of Kubernetes, dynamic admission controllers have long played a pivotal role in enhancing the robustness and adaptability of clusters. For instance, the ValidatingWebhookConfiguration, which empowers users to implement intricate and finely-tuned access controls beyond the capabilities of RBAC. However, this newfound agility often comes at a price – the ease with which they can be misconfigured, potentially leading to cluster disruption and downtime.

Historically, we’ve accepted this fragility as an inevitable trade-off for greater control over our clusters. But what if we could change that narrative?

Enter ValidatingAdmissionPolicies!

In this talk we’ll take a look at what makes ValidatingAdmissionPolicies a safer choice for your validation logic and what problems they aim to solve.

We will delve into the world of ValidatingAdmissionPolicies, exploring their features and limitations. We will also draw comparisons with ValidatingWebhookConfigurations, shedding light on the problems they solve. Furthermore, I’ll provide a comprehensive walkthrough on how you can begin leveraging ValidatingAdmissionPolicies today.

Resources

The following resources were mentioned during the presentation or are useful additional information.

KEP-3488: CEL for Admission Control

Proposal for CEL based admission control (ValidatingAdmissionPolicy)
[KEP 3488] Promote ValidatingAdmissionPolicy to GA

Issue tracking the promotion of ValidatingAdmissionPolicy to GA, currently targetting v1.30
Validating Admission Policy Documentation

The official documentation for using ValidatingAdmissionPolicy
KEP-3962: Mutating Admission Policies

Proposal introducing the concept of MutatingAdmissionPolicy
MutatingAdmissionPolicy implementation PR

The work in progress PR adding the functionality for MutatingAdmissionPolicy
Webhooks - What’s the worst that could happen?

My talk from Rejekts in Chicago about the dangers of admission controller webhooks
Common Expression Language in Kubernetes

The official Kubernetes documentation on CEL, including the custom functions available.
Kyverno - Using CEL expressions

The documentation for using CEL with Kyverno policies.
Tekton - CEL expressions

The documentation for using CEL in Tekton Triggers.
cel-go library

The CEL Go library
CEL Playground

An online, interactive tool for building and testing CEL expressions.

Buzz and feedback

Here’s what was said about this presentation on social media.

Post by @raesene@infosec.exchange

View on Mastodon

From Fragile to Resilient: ValidatingAdmissionPolicies Strengthen Kubernetes

kubectl apply -f deployment.yaml The deployments “nginx-blocked” is invalid: : ValidatingAdmissionPolicy ‘prevent-latest-image-tag’ with binding ‘prevent-latest-image-tag’ denied request: The use of the ‘latest’ tag is not allowed

Link for this presentation:

HTML code for embedding:

Share on social media:

Resources

KEP-3488: CEL for Admission Control

[KEP 3488] Promote ValidatingAdmissionPolicy to GA

Validating Admission Policy Documentation

KEP-3962: Mutating Admission Policies

MutatingAdmissionPolicy implementation PR

Webhooks - What’s the worst that could happen?

Common Expression Language in Kubernetes

Kyverno - Using CEL expressions

Tekton - CEL expressions

cel-go library

CEL Playground

Buzz and feedback