A PeerAuthentication policy in Istio is failing to apply because it’s being overridden by another PeerAuthentication policy with a higher precedence.
Common Causes and Fixes
-
Namespace-Scoped Policy Overriding Global Policy:
- Diagnosis: Check for a
PeerAuthenticationpolicy in the same namespace as your workload or in theistio-systemnamespace. The presence of a policy in the workload’s namespace oristio-systemimplies a higher precedence than a cluster-widemeshpolicy.kubectl get peerauthentication -n <your-workload-namespace> kubectl get peerauthentication -n istio-system - Fix: If you find a conflicting policy (e.g., one in
istio-systemthat you didn’t intend to apply to your workload), you can either:- Delete the conflicting policy if it’s no longer needed:
kubectl delete peerauthentication <conflicting-policy-name> -n <namespace> - Modify the conflicting policy to be less restrictive or to exclude your workload using selectors. For example, to allow permissive mode:
apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: default namespace: istio-system # or your workload namespace spec: mtls: mode: PERMISSIVE
- Delete the conflicting policy if it’s no longer needed:
- Why it works: Istio applies
PeerAuthenticationpolicies based on a precedence order. Namespace-scoped policies (including those inistio-system) have higher precedence than cluster-widemeshpolicies. By removing or modifying the higher-precedence policy, your intended policy can take effect.
- Diagnosis: Check for a
-
Multiple Namespace-Scoped Policies:
- Diagnosis: If you have more than one
PeerAuthenticationpolicy defined within the same namespace, Istio will apply the one with the most specific workload selector. If selectors are identical or absent, it defaults to an undefined order which can lead to unpredictable behavior.
Examine thekubectl get peerauthentication -n <your-workload-namespace> -o yamlspec.selectorfield in each policy. - Fix: Consolidate policies or ensure only one policy applies to your target workload. If multiple policies are genuinely needed, ensure they have distinct selectors that don’t overlap for the same workload. If you need a default for the namespace, ensure one policy has an empty or minimal selector, and others have specific selectors for different workloads.
# Example: A general policy for the namespace (minimal selector) apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: namespace-default namespace: <your-workload-namespace> spec: mtls: mode: STRICT --- # Example: A specific policy for a particular app within the same namespace apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: app-specific-policy namespace: <your-workload-namespace> spec: selector: matchLabels: app: my-special-app mtls: mode: PERMISSIVE - Why it works: Istio prioritizes policies that explicitly select a workload. By having one policy with a broad or no selector for the namespace’s default, and others with specific selectors for particular applications, you create a clear hierarchy. The most specific policy wins for a given workload.
- Diagnosis: If you have more than one
-
Conflicting Workload Selectors in the Same Namespace:
- Diagnosis: You have two or more
PeerAuthenticationpolicies in the same namespace, and theirspec.selectorfields match the labels of the same workload.
Look for policies wherekubectl get peerauthentication -n <your-workload-namespace> -o yamlspec.selector.matchLabelsorspec.selector.matchExpressionsare identical and apply to your pod. - Fix: Remove or modify one of the conflicting policies. If you need different mTLS settings for different groups of pods within the same namespace, ensure their selectors are mutually exclusive.
# Policy 1 (for specific pods) apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: policy-for-pods-a namespace: <your-workload-namespace> spec: selector: matchLabels: app: my-app tier: frontend mtls: mode: STRICT --- # Policy 2 (for other specific pods) apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: policy-for-pods-b namespace: <your-workload-namespace> spec: selector: matchLabels: app: my-app tier: backend mtls: mode: PERMISSIVE - Why it works: When multiple policies in the same namespace select the same workload, Istio’s behavior is undefined. By ensuring selectors are distinct and do not overlap for any given workload, you eliminate this ambiguity, allowing the intended policy to be applied.
- Diagnosis: You have two or more
-
Global
meshPolicy as the Intended Default:- Diagnosis: You expect a cluster-wide
meshpolicy to be active, but it’s not taking effect. This often happens if a namespace-scoped policy (even an empty one) exists and has higher precedence.
Check if anykubectl get peerauthentication -n <your-workload-namespace> kubectl get peerauthentication -n istio-system kubectl get peerauthentication --all-namespaces | grep meshPeerAuthenticationexists inistio-systemor your workload’s namespace. - Fix: If a namespace-scoped policy is present and you intend for the
meshpolicy to be the effective default, delete the namespace-scoped policy.
Ensure yourkubectl delete peerauthentication <namespace-scoped-policy-name> -n <namespace>meshpolicy is correctly configured, for example, to enforce STRICT mTLS:apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: default namespace: istio-system spec: mtls: mode: STRICT - Why it works: Namespace-scoped policies (including those in
istio-system) always take precedence overmesh(cluster-scoped) policies. Removing the interfering namespace-scoped policy allows themeshpolicy to be the active one for workloads that are not covered by any other specific namespace-scoped policy.
- Diagnosis: You expect a cluster-wide
-
Incorrectly Applied
meshPolicy (e.g., wrong name or namespace):- Diagnosis: You’ve created a
PeerAuthenticationpolicy withnamespace: istio-systemandname: default, but it’s not being enforced. Double-check themetadata.nameandmetadata.namespacefields. The onlyPeerAuthenticationpolicy that applies cluster-wide to all workloads (unless overridden by a namespace-scoped policy) must be nameddefaultand reside in theistio-systemnamespace.kubectl get peerauthentication default -n istio-system -o yaml - Fix: Ensure your global
PeerAuthenticationpolicy is nameddefaultand is in theistio-systemnamespace. If it has a different name or is in another namespace, delete it and recreate it with the correct name and namespace.# Delete if it exists with the wrong name/namespace kubectl delete peerauthentication <wrong-name> -n <wrong-namespace> # Create the correct one cat <<EOF | kubectl apply -f - apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: default namespace: istio-system spec: mtls: mode: STRICT # Or PERMISSIVE, DISABLE EOF - Why it works: Istio specifically looks for a
PeerAuthenticationresource nameddefaultin theistio-systemnamespace to apply as the mesh-wide policy. Any other name or namespace will cause it to be treated as a regular namespace-scoped policy, subject to selector matching and precedence rules.
- Diagnosis: You’ve created a
-
Mismatched
spec.selectorLabels:- Diagnosis: You have a
PeerAuthenticationpolicy with aspec.selectordefined, but the labels on your workload pods do not match the selector.
Compare the# Check your PeerAuthentication policy kubectl get peerauthentication <policy-name> -n <namespace> -o yaml # Check the labels on your workload pods kubectl get pods -n <namespace> --show-labelsspec.selector.matchLabelsorspec.selector.matchExpressionswith the output ofkubectl get pods --show-labels. - Fix: Update either the policy’s selector or the workload’s labels to ensure they match.
If your pods have labels# Policy with specific selector apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: my-app-policy namespace: <your-workload-namespace> spec: selector: matchLabels: app: my-application # This label must exist on the pods version: v1 mtls: mode: STRICTapp: my-applicationandversion: v1, this policy will apply. If they have different labels, it won’t. - Why it works: A
PeerAuthenticationpolicy with aspec.selectoronly applies to workloads (pods) whose labels precisely match the selector. If there’s a mismatch, the policy is effectively ignored for that workload, and Istio falls back to the next applicable policy based on precedence.
- Diagnosis: You have a
After resolving these conflicts, the next error you might encounter is related to authorization policies (AuthorizationPolicy) not being able to enforce rules due to mTLS being in PERMISSIVE mode when STRICT was expected by the authorization policy.