Linkerd injects its proxy automatically, but sometimes it doesn’t, and you’re left wondering why your services aren’t meshed.
This happens because the Linkerd control plane relies on specific annotations on your Kubernetes namespaces and pods to know what to inject. If those annotations are missing or incorrect, the injection process simply skips that resource.
Common Causes and Fixes
1. Namespace Not Annotated for Injection
-
Diagnosis: Check if your target namespace has the
linkerd.io/inject: enabledannotation.kubectl get namespace <your-namespace> -o yamlLook for
metadata.annotations.linkerd.io/inject: enabled. -
Fix: Add the annotation to the namespace.
kubectl annotate namespace <your-namespace> linkerd.io/inject=enabledThis tells Linkerd’s admission controller to intercept all new pods created in this namespace and attempt injection.
-
Why it works: Linkerd’s mutating admission webhook watches for this annotation on namespaces. When a new pod is created in an annotated namespace, the webhook is triggered, and it adds the Linkerd proxy container to the pod’s definition.
2. Pod Explicitly Disabled for Injection
-
Diagnosis: Even if the namespace is annotated, a pod can opt-out. Check the pod’s annotations for
linkerd.io/inject: disabled.kubectl get pod <your-pod-name> -n <your-namespace> -o yamlLook for
metadata.annotations.linkerd.io/inject: disabled. -
Fix: Remove the
linkerd.io/inject: disabledannotation from the pod.kubectl annotate pod <your-pod-name> -n <your-namespace> linkerd.io/inject-The trailing hyphen (
-) removes the annotation. -
Why it works: The admission webhook checks for an explicit disable annotation on the pod after checking the namespace. If found, it skips injection for that specific pod, overriding the namespace-level setting.
3. Pod Created Before Namespace Annotation
-
Diagnosis: If you enabled injection on a namespace after pods were already running, those existing pods won’t be automatically injected. You’d see the namespace annotated but not the pods.
-
Fix: Delete and recreate the pods in the namespace after the namespace has been annotated for injection.
kubectl delete pod <your-pod-name> -n <your-namespace> # Linkerd will automatically inject the proxy when the pod is recreatedAlternatively, if using a Deployment, you can trigger a rollout restart.
kubectl rollout restart deployment <your-deployment-name> -n <your-namespace> -
Why it works: Linkerd’s injection is a mutating admission webhook process that only intercepts new pod creations. Existing pods don’t get re-evaluated for injection unless they are deleted and recreated.
4. Incorrect Linkerd Version or Installation Issues
-
Diagnosis: Ensure your Linkerd control plane is healthy and that the version is compatible with your Kubernetes version. Check the Linkerd control plane pods for errors.
kubectl get pods -n linkerd -o wide kubectl logs <linkerd-controller-pod-name> -n linkerd kubectl logs <linkerd-admission-webhooks-pod-name> -n linkerdLook for errors related to admission control or resource creation.
-
Fix: If the control plane is unhealthy, troubleshoot those pods. If it’s an old version, upgrade Linkerd according to the official documentation.
# Example: Upgrade Linkerd CLI and apply new control plane manifests # (Refer to Linkerd official upgrade docs for specific commands) linkerd upgrade --from-version <old-version> | kubectl apply -f - -
Why it works: The admission webhook, responsible for injecting the proxy, is part of the Linkerd control plane. If the control plane is not running correctly, or if there are compatibility issues, the injection mechanism will fail.
5. Resource Limits on Admission Webhook Pods
-
Diagnosis: The Linkerd admission webhook pods might be crashing or not starting due to insufficient CPU or memory. Check their status and logs.
kubectl get pods -n linkerd -l app=linkerd-admission-webhook kubectl describe pod <linkerd-admission-webhook-pod-name> -n linkerd kubectl logs <linkerd-admission-webhook-pod-name> -n linkerdLook for OOMKilled (Out Of Memory) or CPU throttling messages.
-
Fix: Increase the resource requests and limits for the admission webhook pods in the Linkerd control plane configuration. This is typically done by updating the
linkerd-control-plane.yamlor similar manifest before applying it.# Example snippet from linkerd-control-plane.yaml (values are illustrative) apiVersion: apps/v1 kind: Deployment metadata: name: linkerd-admission-webhook namespace: linkerd spec: template: spec: containers: - name: webhook resources: requests: cpu: 100m memory: 128Mi limits: cpu: 200m memory: 256MiApply the updated manifest:
kubectl apply -f linkerd-control-plane.yaml -
Why it works: The admission webhook needs sufficient resources to process incoming pod creation requests. If it’s starved for CPU or memory, it can become unresponsive or crash, preventing injection.
6. Pods Without serviceAccountName (Less Common for Injection)
-
Diagnosis: While not directly a cause of injection failure, pods that don’t have a
serviceAccountNamemight exhibit strange behavior later, including not being able to communicate after injection. If you’re seeing injection but then no service communication, check this.kubectl get pod <your-pod-name> -n <your-namespace> -o yamlLook for
spec.serviceAccountName. -
Fix: Assign a ServiceAccount to your pod or deployment.
# In your pod or deployment spec spec: serviceAccountName: <your-service-account-name>Ensure the ServiceAccount exists and has appropriate RBAC permissions.
-
Why it works: The Linkerd proxy uses the pod’s ServiceAccount to make Kubernetes API calls (e.g., for service discovery, getting pod information). Without one, it lacks the necessary identity and permissions.
After ensuring your namespaces and pods are correctly annotated and your Linkerd control plane is healthy, you might encounter issues with the Linkerd Dashboard not showing your services if the linkerd-viz component is not installed or healthy.