Kubernetes admission controllers are the gatekeepers of your cluster, but they don’t just say "yes" or "no" to requests; they can actively change them before they even hit the API server’s core logic.
Let’s see this in action. Imagine you want to ensure every Pod created in your cluster has a specific label, say team: backend. Without an admission controller, you’d have to rely on developers remembering to add it, or run a separate process to find and label Pods later.
Here’s a simple Pod definition that doesn’t have the label:
apiVersion: v1
kind: Pod
metadata:
name: my-app
spec:
containers:
- name: main
image: nginx:latest
Now, let’s say we have a ValidatingWebhookConfiguration and a MutatingWebhookConfiguration set up. When this Pod definition is sent to the Kubernetes API server, it first passes through the validating webhooks. If they all approve, it then goes to the mutating webhooks. A mutating webhook could intercept this request and add the team: backend label. The API server then proceeds with the modified request, and the Pod is created with the label already in place.
apiVersion: v1
kind: Pod
metadata:
name: my-app
labels:
team: backend # This label was added by a mutating webhook
spec:
containers:
- name: main
image: nginx:latest
Admission controllers work by intercepting requests to the Kubernetes API server after authentication and authorization but before the request is persisted to etcd. They are implemented as webhooks that the API server can call out to. There are two main types:
- Validating Admission Webhooks: These controllers examine the incoming request and can either allow it to proceed or reject it with an error. They don’t modify the request. Their primary purpose is to enforce policies and prevent invalid configurations from entering the cluster.
- Mutating Admission Webhooks: These controllers can intercept requests, modify them (e.g., add labels, inject sidecar containers, set default resource limits), and then allow them to proceed. They are crucial for automating common configurations and ensuring consistency.
The API server has a built-in list of admission controllers that are always enabled and run in a specific order. However, for custom logic, you define your own admission controllers using MutatingWebhookConfiguration and ValidatingWebhookConfiguration resources. These resources tell the API server which endpoints (your webhook services) to call for specific operations (CREATE, UPDATE, DELETE, CONNECT) on specific resources (Pods, Deployments, etc.).
Let’s dive into the mental model of how these webhooks are triggered and processed. When a request hits the API server, it first consults its internal, built-in admission controllers. If all of those pass, it then looks at the MutatingWebhookConfiguration and ValidatingWebhookConfiguration resources.
For mutating webhooks, the API server sends the request to the specified webhook service(s) for any matching configurations. If multiple mutating webhooks match, they are called in the order they appear in the MutatingWebhookConfiguration list. The response from each webhook can modify the object. The API server then re-evaluates the object against any subsequent webhooks in the list, which will see the modifications made by the earlier ones. This chaining is powerful, allowing for complex, layered transformations.
After all mutating webhooks have run, the (potentially modified) object is then passed to the validating webhooks. Again, if multiple validating webhooks match, they are called sequentially. If any validating webhook rejects the request, the entire request is denied, and the object is not persisted.
The failurePolicy in your webhook configurations is critical. If set to Fail, any error or timeout from your webhook service will cause the API server to reject the admission request. If set to Ignore, the API server will proceed as if the webhook wasn’t there, allowing requests even if your webhook fails.
Consider the namespaceSelector and objectSelector fields in your webhook configurations. These are incredibly powerful for fine-grained control. You can specify that a webhook should only apply to resources in certain namespaces (e.g., namespaceSelector: { matchLabels: { "admission-control-enabled": "true" } }) or to objects with specific labels. This prevents your webhooks from interfering with system namespaces or specific application deployments unless intended.
The most surprising thing is how the API server handles the request lifecycle between mutating and validating webhooks. After a mutating webhook modifies an object, the API server doesn’t just send the modified object to the next mutating webhook. Instead, it effectively re-evaluates the entire request, including the modified object, against the entire set of configured webhooks (both mutating and validating) as if it were a new request. This means a mutation could potentially cause a previously-validating webhook to now fail, or a previously-failing webhook to now pass. This re-evaluation loop is key to understanding why complex webhook chains behave as they do.
The next logical step is to explore how to implement these webhook services yourself, often using frameworks like kube-webhook-certgen to handle TLS certificates automatically.