The Future of Kubernetes Admission Logic

Purpose / Use Cases Defaulting Policy Enforcement ● Injecting imagePullSecrets dynamically when pods are created ● Injecting sidecars into pods ● Injecting proxy environment variables into pods ● Prevent using latest image tag ● Require all pods to have resource limits set ● Block the use of deprecated Kubernetes APIs (e.g. batch/v1beta1) ● Require a PodDisruptionBudget ● Enforce a standard set of labels / annotations on all resources ● Replace all image registries with an in-house container image proxy / cache ● Block nodes joining the cluster with known CVEs based on the kernel version (e.g. CVE-2022-0185) ● Inject Log4Shell mitigation env var into all pods (CVE-2021-44228) ● Block binding to the cluster-admin role Best Practices Problem Mitigation 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Brief introduction to CEL ● Uses a similar syntax to the expressions in C-based languages, e.g. self.minReplicas <= self.replicas && self.replicas <= self.maxReplicas ● Designed to be embedded into other applications with a focus on “one-liners” of code. ● Small number of built in functions (e.g. split, has) ● Functions expanded through custom libraries (Kubernetes includes several of these) References: ● https://kubernetes.io/docs/reference/usingapi/cel/ ● https://github.com/google/cel-go ● https://playcel.undistro.io/ (CEL playground) ● Already used by Kyverno, Tekton, etc. ● Has a concept of “cost” of expressions that helps ensure safe, side-effect free running of untrusted code. 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example Blocking the use of the ‘latest’ image tag prevent-latest-vap.yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: “prevent-latest-image-tag” spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”, “UPDATE”] resources: [“deployments”, “daemonsets”, “statefulsets”] validations: - message: “The use of the ‘latest’ tag is not allowed” expression: | object.spec.template.spec.containers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) && ( !has(object.spec.template.spec.initContainers) || object.spec.template.spec.initContainers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) ) More examples: github.com/AverageMarcus/comm on-admission-policies 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example Blocking the use of the ‘latest’ image tag prevent-latest-vap.yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: “prevent-latest-image-tag” spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”, “UPDATE”] resources: [“deployments”, “daemonsets”, “statefulsets”] validations: - message: “The use of the ‘latest’ tag is not allowed” expression: | object.spec.template.spec.containers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) && ( !has(object.spec.template.spec.initContainers) || object.spec.template.spec.initContainers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) ) The name of our policy. We’ll reference this in our ValidatingAdmission PolicyBinding 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example Blocking the use of the ‘latest’ image tag prevent-latest-vap.yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: “prevent-latest-image-tag” spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”, “UPDATE”] resources: [“deployments”, “daemonsets”, “statefulsets”] validations: - message: “The use of the ‘latest’ tag is not allowed” expression: | object.spec.template.spec.containers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) && ( !has(object.spec.template.spec.initContainers) || object.spec.template.spec.initContainers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) ) How we should handle mistakes / runtime errors in our CEL expressions. By default it will block the request. The alternative is Ignore if you want to allow requests on error. Note: This is when something errors, not when the validations fail. E.g. typo in CEL expression 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example Blocking the use of the ‘latest’ image tag prevent-latest-vap.yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: “prevent-latest-image-tag” spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”, “UPDATE”] resources: [“deployments”, “daemonsets”, “statefulsets”] validations: - message: “The use of the ‘latest’ tag is not allowed” expression: | object.spec.template.spec.containers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) && ( !has(object.spec.template.spec.initContainers) || object.spec.template.spec.initContainers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) ) We define what resources this policy applies to. Multiple resource types can be defined here but if they don’t have the same general API the CEL expression will quickly become very complex. Supported operations are: CREATE, UPDATE, DELETE, CONNECT or * (meaning all the previous) 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example Blocking the use of the ‘latest’ image tag prevent-latest-vap.yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: “prevent-latest-image-tag” spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”, “UPDATE”] resources: [“deployments”, “daemonsets”, “statefulsets”] validations: - message: “The use of the ‘latest’ tag is not allowed” expression: | object.spec.template.spec.containers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) && ( !has(object.spec.template.spec.initContainers) || object.spec.template.spec.initContainers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) ) Validations define one or more conditions that must be met for the request to be allowed. The array is AND’d together and all must pass. 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example Blocking the use of the ‘latest’ image tag prevent-latest-vap.yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: “prevent-latest-image-tag” spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”, “UPDATE”] resources: [“deployments”, “daemonsets”, “statefulsets”] validations: - message: “The use of the ‘latest’ tag is not allowed” expression: | object.spec.template.spec.containers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) && ( !has(object.spec.template.spec.initContainers) || object.spec.template.spec.initContainers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) ) We’ll add a human-friendly message to be shown when the API request has been blocked by this policy. If we don’t include this, a generic message that include the whole expression is shown instead. 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example Blocking the use of the ‘latest’ image tag prevent-latest-vap.yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: “prevent-latest-image-tag” spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”, “UPDATE”] resources: [“deployments”, “daemonsets”, “statefulsets”] validations: - message: “The use of the ‘latest’ tag is not allowed” expression: | object.spec.template.spec.containers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) && ( !has(object.spec.template.spec.initContainers) || object.spec.template.spec.initContainers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) ) Finally we have our expression. This is the expression of allowed resources, not those to block. (I keep getting caught out by this 😅) 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example Blocking the use of the ‘latest’ image tag prevent-latest-vap.yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: “prevent-latest-image-tag” spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”, “UPDATE”] resources: [“deployments”, “daemonsets”, “statefulsets”] validations: - message: “The use of the ‘latest’ tag is not allowed” expression: | object.spec.template.spec.containers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) && ( !has(object.spec.template.spec.initContainers) || object.spec.template.spec.initContainers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) ) object is the incoming resource from the API call. 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example Blocking the use of the ‘latest’ image tag prevent-latest-vap.yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: “prevent-latest-image-tag” spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”, “UPDATE”] resources: [“deployments”, “daemonsets”, “statefulsets”] validations: - message: “The use of the ‘latest’ tag is not allowed” expression: | object.spec.template.spec.containers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) && ( !has(object.spec.template.spec.initContainers) || object.spec.template.spec.initContainers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) ) First we check all containers don’t have the latest tag (or no tag specified). 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example Blocking the use of the ‘latest’ image tag prevent-latest-vap.yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: “prevent-latest-image-tag” spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [“apps”] apiVersions: [“v1”] operations: [“CREATE”, “UPDATE”] resources: [“deployments”, “daemonsets”, “statefulsets”] validations: - message: “The use of the ‘latest’ tag is not allowed” expression: | object.spec.template.spec.containers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) && ( !has(object.spec.template.spec.initContainers) || object.spec.template.spec.initContainers.all(container, !container.image.endsWith(“:latest”) && container.image.contains(“:”) ) ) Then we check if this resource defines initContainers and if so we check they also don’t use latest. 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

More advanced features ● More context values - Along with object you also have oldObject, namespaceObject & request you can use in your expressions ● Parameters - make your policies configurable by allowing parameter resources to be applied by the binding policy.yaml policy-binding.yaml apiVersion: admissionregistration.k8s… kind: ValidatingAdmissionPolicy metadata: name: “param-example” spec: paramKind: apiVersion: rules.example.com/v1 kind: ReplicaLimit validations: - expression: object.spec.replicas <= params.maxReplicas apiVersion: admissionregistration.k8s… kind: ValidatingAdmissionPolicyBinding metadata: name: “param-example” spec: policyName: “param-example” paramRef: name: “my-parameters” namespace: “default” params.yaml apiVersion: rules.example.com/v1 kind: ReplicaLimit metadata: name: “my-parameters” namespace: “default” maxReplicas: 3 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

More advanced features ● More context values - Along with object you also have oldObject, namespaceObject & request you can use in your expressions ● Parameters - make your policies configurable by allowing parameter resources to be applied by the binding ● Variables - reusable CEL expressions to simplify your validation expressions. ● Audit annotations - Add extra metadata to the audit logs, dynamic values using CEL auditAnnotations: - key: “replica-count” valueExpression: | “‘Deployment spec.replicas set to ’ + string(object.spec.replicas)” { “kind”: “Event”, “apiVersion”: “audit.k8s.io/v1”, “annotations”: { “demo-policy.example.com/replica-count”: “Deployment spec.replicas set to 128” … } } 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

MutatingAdmissionPolicies Status: Alpha in v1.32 Introduced in: KEP-3962 ● Follows the same idea as ValidatingAdmissionPolicies. ● Introduces MutatingAdmissionPolicy and MutatingAdmissionPolicyBinding ● Supports two patch strategies: ApplyConfiguration and JSONPatch ● Not yet enabled by default, you must enable the following: ○ MutatingAdmissionPolicy feature gate ○ admissionregistration.k8s.io/v1alpha1 API kind-config.yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: kind-cluster featureGates: “MutatingAdmissionPolicy”: true runtimeConfig: “admissionregistration.k8s.io/v1alpha1”: true Create a Kind cluster with: kind create cluster —config kind-config.yaml 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example proxy-vars-map.yaml Injecting proxy values as env vars Same as ValidatingAdmissionPolicies apiVersion: admissionregistration.k8s.io/v1alpha1 kind: MutatingAdmissionPolicy metadata: name: “proxy-values” spec: failurePolicy: Fail reinvocationPolicy: IfNeeded matchConstraints: resourceRules: - apiGroups: [“”] apiVersions: [“v1”] operations: [“CREATE”] resources: [“pods”] mutations: - patchType: “ApplyConfiguration” applyConfiguration: > expression: > Object{ spec: Object.spec{ containers: object.spec.containers.map(c, Object.spec.containers.item{ name: c.name, env: [ Object.spec.containers.env{ name: “HTTP_PROXY”, value: “http://proxy.proxy.svc:3128” } ] } ) } } 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example proxy-vars-map.yaml Injecting proxy values as env vars Should the policy be re-applied if other mutations are made to the resource? Allowed values are “Never” and “IfNeeded” apiVersion: admissionregistration.k8s.io/v1alpha1 kind: MutatingAdmissionPolicy metadata: name: “proxy-values” spec: failurePolicy: Fail reinvocationPolicy: IfNeeded matchConstraints: resourceRules: - apiGroups: [“”] apiVersions: [“v1”] operations: [“CREATE”] resources: [“pods”] mutations: - patchType: “ApplyConfiguration” applyConfiguration: > expression: > Object{ spec: Object.spec{ containers: object.spec.containers.map(c, Object.spec.containers.item{ name: c.name, env: [ Object.spec.containers.env{ name: “HTTP_PROXY”, value: “http://proxy.proxy.svc:3128” } ] } ) } } 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example proxy-vars-map.yaml Injecting proxy values as env vars Replace validations with mutations apiVersion: admissionregistration.k8s.io/v1alpha1 kind: MutatingAdmissionPolicy metadata: name: “proxy-values” spec: failurePolicy: Fail reinvocationPolicy: IfNeeded matchConstraints: resourceRules: - apiGroups: [“”] apiVersions: [“v1”] operations: [“CREATE”] resources: [“pods”] mutations: - patchType: “ApplyConfiguration” applyConfiguration: > expression: > Object{ spec: Object.spec{ containers: object.spec.containers.map(c, Object.spec.containers.item{ name: c.name, env: [ Object.spec.containers.env{ name: “HTTP_PROXY”, value: “http://proxy.proxy.svc:3128” } ] } ) } } 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example proxy-vars-map.yaml Injecting proxy values as env vars Indicates the patch strategy Possible values: ● ApplyConfiguration ● JSONPatch apiVersion: admissionregistration.k8s.io/v1alpha1 kind: MutatingAdmissionPolicy metadata: name: “proxy-values” spec: failurePolicy: Fail reinvocationPolicy: IfNeeded matchConstraints: resourceRules: - apiGroups: [“”] apiVersions: [“v1”] operations: [“CREATE”] resources: [“pods”] mutations: - patchType: “ApplyConfiguration” applyConfiguration: > expression: > Object{ spec: Object.spec{ containers: object.spec.containers.map(c, Object.spec.containers.item{ name: c.name, env: [ Object.spec.containers.env{ name: “HTTP_PROXY”, value: “http://proxy.proxy.svc:3128” } ] } ) } } 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example proxy-vars-map.yaml Injecting proxy values as env vars Either applyConfiguration or jsonPatch depending on the patchType. apiVersion: admissionregistration.k8s.io/v1alpha1 kind: MutatingAdmissionPolicy metadata: name: “proxy-values” spec: failurePolicy: Fail reinvocationPolicy: IfNeeded matchConstraints: resourceRules: - apiGroups: [“”] apiVersions: [“v1”] operations: [“CREATE”] resources: [“pods”] mutations: - patchType: “ApplyConfiguration” applyConfiguration: > expression: > Object{ spec: Object.spec{ containers: object.spec.containers.map(c, Object.spec.containers.item{ name: c.name, env: [ Object.spec.containers.env{ name: “HTTP_PROXY”, value: “http://proxy.proxy.svc:3128” } ] } ) } } 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example proxy-vars-map.yaml Injecting proxy values as env vars Introduction of named types The Object refers to the type of the incoming resource (Pod in this example). Child fields can be used by using Object.[fieldname] and can go multiple levels down as see here with the Object.spec.containers.env apiVersion: admissionregistration.k8s.io/v1alpha1 kind: MutatingAdmissionPolicy metadata: name: “proxy-values” spec: failurePolicy: Fail reinvocationPolicy: IfNeeded matchConstraints: resourceRules: - apiGroups: [“”] apiVersions: [“v1”] operations: [“CREATE”] resources: [“pods”] mutations: - patchType: “ApplyConfiguration” applyConfiguration: > expression: > Object{ spec: Object.spec{ containers: object.spec.containers.map(c, Object.spec.containers.item{ name: c.name, env: [ Object.spec.containers.env{ name: “HTTP_PROXY”, value: “http://proxy.proxy.svc:3128” } ] } ) } } 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social

Example proxy-vars-map.yaml Injecting proxy values as env vars Merge our proxy vars with any existing vars in all containers. As this is an ApplyConfiguration patch type the existing env vars on the object remain untouched. Note: We’re missing initContainers and ephermeralContainers here due to limited space. apiVersion: admissionregistration.k8s.io/v1alpha1 kind: MutatingAdmissionPolicy metadata: name: “proxy-values” spec: failurePolicy: Fail reinvocationPolicy: IfNeeded matchConstraints: resourceRules: - apiGroups: [“”] apiVersions: [“v1”] operations: [“CREATE”] resources: [“pods”] mutations: - patchType: “ApplyConfiguration” applyConfiguration: > expression: > Object{ spec: Object.spec{ containers: object.spec.containers.map(c, Object.spec.containers.item{ name: c.name, env: [ Object.spec.containers.env{ name: “HTTP_PROXY”, value: “http://proxy.proxy.svc:3128” } ] } ) } } 🐘 @Marcus@k8s.social | 🌐 MarcusNoble.com | 🦋 @averagemarcus.bsky.social