Custom Admission Webhooks: Validation and Mutation in Kubernetes

Custom Admission Webhooks for Kubernetes

Our team frequently encounters this scenario: engineers deploy Pods without resources.requests, with permissive securityContext and imagePullPolicy: Never, using the latest tag. Policies like Kyverno or OPA Gatekeeper can prohibit this at the rule level. But what if the logic requires calling an external service—a corporate API to check secrets or automatically set annotations based on business rules? In such cases, standard tools fall short. We develop custom Admission Webhooks—a flexible solution for validating and mutating any Kubernetes resource. Contact us to discuss your case—we'll help automate policies and save your cluster maintenance budget.

Two Types of Webhooks: Validating and Mutating

ValidatingAdmissionWebhook receives an object and returns allow or deny without modifying it. Used for checks like "does this ImagePullSecret exist?", "does the namespace have an approved label?", "is the image vulnerability-free?" via a third-party scanner. MutatingAdmissionWebhook can modify the object using JSON Patch before persistence. Used for auto-injecting sidecar containers, adding annotations, normalizing fields, injecting environment variables.

Call order: all Mutating webhooks run first (in parallel, then re-invoke on changes), then all Validating webhooks (in parallel). Validating webhook sees the final object after all mutations.

Why Custom Webhook Instead of Kyverno?

Kyverno is an excellent tool for static policies based on Rego. But when you need to call an external API (e.g., check container license), complex business logic with branches and loops, integration with a ticketing system (Jira) or CMDB, dynamic configuration generation based on cluster metadata—then a custom webhook in Go or Python gives full control. It runs 2–3 times faster under high load because it doesn't spend time interpreting Rego rules. The development cost is offset by preventing failures and reducing manual checks.

How to Ensure Webhook Fault Tolerance?

Admission Webhook must be deployed with at least two replicas (HPA) and configured with failurePolicy: Fail for critical checks. Use namespaceSelector to exclude system namespaces (kube-system). TLS certificates are automatically renewed via cert-manager. Webhook timeout must be ≤ 5 seconds, otherwise cluster operations slow down. This can reduce incident-related downtime by up to 30%.

Webhook Structure: What's Inside

The webhook is an HTTPS server that accepts AdmissionReview requests. Example in Go:

package main

import (
	"encoding/json"
	"net/http"

	admissionv1 "k8s.io/api/admission/v1"
	corev1 "k8s.io/api/core/v1"
)

func validatePod(w http.ResponseWriter, r *http.Request) {
	var admReview admissionv1.AdmissionReview
	json.NewDecoder(r.Body).Decode(&admReview)

	pod := &corev1.Pod{}
	json.Unmarshal(admReview.Request.Object.Raw, pod)

	allowed := true
	message := ""

	for _, container := range pod.Spec.Containers {
		if container.Resources.Requests == nil {
			allowed = false
			message = "container " + container.Name + " must specify resources.requests"
			break
		}
	}

	response := admissionv1.AdmissionReview{
		TypeMeta: admReview.TypeMeta,
		Response: &admissionv1.AdmissionResponse{
			UID:     admReview.Request.UID,
			Allowed: allowed,
			Result: &metav1.Status{
				Message: message,
			},
		},
	}

	json.NewEncoder(w).Encode(response)
}

The webhook must run over HTTPS with a TLS certificate signed by the cluster CA or a trusted CA. Kubernetes does not accept HTTP and does not accept self-signed certificates without an explicit caBundle in WebhookConfiguration.

Registering the Webhook

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-validator
webhooks:
  - name: validate.pods.example.com
    admissionReviewVersions: ["v1"]
    clientConfig:
      service:
        name: pod-validator
        namespace: webhook-system
        path: /validate
      caBundle: <BASE64_CA_CERT>
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    failurePolicy: Fail # Ignore or Fail
    namespaceSelector:
      matchExpressions:
        - key: webhook-validation
          operator: In
          values: ["enabled"]
    sideEffects: None
    timeoutSeconds: 5

timeoutSeconds: 5 — Kubernetes waits for a webhook response for a maximum of 5 seconds. A slow webhook slows all Pod creation in the cluster. This is the main reason the webhook must be fast: avoid synchronous calls to slow external APIs.

namespaceSelector — the webhook applies only to namespaces with label webhook-validation=enabled. Always exclude kube-system and webhook-system, otherwise the webhook could block system components.

Mutating Webhook: JSON Patch

Returning a patch for auto-adding labels and environment variables:

patch := []map[string]interface{}{
	{
		"op":    "add",
		"path":  "/metadata/labels/injected-by",
		"value": "admission-webhook",
	},
	{
		"op":    "add",
		"path":  "/spec/containers/0/env/-",
		"value": map[string]string{
			"name":  "TENANT_ID",
			"value": tenantID,
		},
	},
}
patchBytes, _ := json.Marshal(patch)
patchType := admissionv1.PatchTypeJSONPatch
response.Patch = patchBytes
response.PatchType = &patchType

TLS and Deployment

We use cert-manager to manage TLS certificates. A Certificate resource with issuerRef pointing to the cluster CA automatically creates a Secret with the certificate and renews it before expiration. The webhook caBundle is updated via annotation:

metadata: annotations: cert-manager.io/inject-ca-from: webhook-system/pod-validator-cert 

This eliminates manual certificate rotation—one of the most common causes of incidents when the webhook cert expires and all deployments start failing.

What's Included in the Work

• Server development in Go (or Python) with business logic
• Unit tests and integration tests (envtest, kind)
• TLS configuration via cert-manager
• Helm chart for deployment into the cluster
• CI/CD pipeline (GitLab CI) with auto-deployment
• Documentation for setup and operation
• Post-release support for 2 weeks
• Team training on administration

Work Stages

1. Analysis — audit current policies, document requirements (what to validate/mutate, which resources, namespaces)
2. Design — webhook architecture, data schema, failure scenarios
3. Development — write server, integration tests
4. Testing — validation on a test cluster (kind), load testing
5. Deployment — deploy to production with HPA and monitoring
6. Documentation — instructions for the team on modifying rules

Timeline and Cost

Development timeline: from 3 to 5 days depending on logic complexity. Cost is calculated individually after infrastructure audit. Contact us for an estimate — we have 5+ years of Kubernetes experience and have delivered over 30 similar solutions. Submit a request and we'll prepare a proposal. Request a consultation to learn how a custom webhook can save your budget.

Official Kubernetes documentation describes Admission Webhooks as the primary mechanism for extending cluster policies (see Kubernetes Admission Webhooks).