A PeerAuthentication policy in Istio is failing to apply because it’s being overridden by another PeerAuthentication policy with a higher precedence.

Common Causes and Fixes

  1. Namespace-Scoped Policy Overriding Global Policy:

    • Diagnosis: Check for a PeerAuthentication policy in the same namespace as your workload or in the istio-system namespace. The presence of a policy in the workload’s namespace or istio-system implies a higher precedence than a cluster-wide mesh policy.
      kubectl get peerauthentication -n <your-workload-namespace>
      kubectl get peerauthentication -n istio-system
      
    • Fix: If you find a conflicting policy (e.g., one in istio-system that you didn’t intend to apply to your workload), you can either:
      • Delete the conflicting policy if it’s no longer needed:
        kubectl delete peerauthentication <conflicting-policy-name> -n <namespace>
        
      • Modify the conflicting policy to be less restrictive or to exclude your workload using selectors. For example, to allow permissive mode:
        apiVersion: security.istio.io/v1beta1
        kind: PeerAuthentication
        metadata:
          name: default
          namespace: istio-system # or your workload namespace
        spec:
          mtls:
            mode: PERMISSIVE
        
    • Why it works: Istio applies PeerAuthentication policies based on a precedence order. Namespace-scoped policies (including those in istio-system) have higher precedence than cluster-wide mesh policies. By removing or modifying the higher-precedence policy, your intended policy can take effect.
  2. Multiple Namespace-Scoped Policies:

    • Diagnosis: If you have more than one PeerAuthentication policy defined within the same namespace, Istio will apply the one with the most specific workload selector. If selectors are identical or absent, it defaults to an undefined order which can lead to unpredictable behavior.
      kubectl get peerauthentication -n <your-workload-namespace> -o yaml
      
      Examine the spec.selector field in each policy.
    • Fix: Consolidate policies or ensure only one policy applies to your target workload. If multiple policies are genuinely needed, ensure they have distinct selectors that don’t overlap for the same workload. If you need a default for the namespace, ensure one policy has an empty or minimal selector, and others have specific selectors for different workloads.
      # Example: A general policy for the namespace (minimal selector)
      apiVersion: security.istio.io/v1beta1
      kind: PeerAuthentication
      metadata:
        name: namespace-default
        namespace: <your-workload-namespace>
      spec:
        mtls:
          mode: STRICT
      ---
      # Example: A specific policy for a particular app within the same namespace
      apiVersion: security.istio.io/v1beta1
      kind: PeerAuthentication
      metadata:
        name: app-specific-policy
        namespace: <your-workload-namespace>
      spec:
        selector:
          matchLabels:
            app: my-special-app
        mtls:
          mode: PERMISSIVE
      
    • Why it works: Istio prioritizes policies that explicitly select a workload. By having one policy with a broad or no selector for the namespace’s default, and others with specific selectors for particular applications, you create a clear hierarchy. The most specific policy wins for a given workload.
  3. Conflicting Workload Selectors in the Same Namespace:

    • Diagnosis: You have two or more PeerAuthentication policies in the same namespace, and their spec.selector fields match the labels of the same workload.
      kubectl get peerauthentication -n <your-workload-namespace> -o yaml
      
      Look for policies where spec.selector.matchLabels or spec.selector.matchExpressions are identical and apply to your pod.
    • Fix: Remove or modify one of the conflicting policies. If you need different mTLS settings for different groups of pods within the same namespace, ensure their selectors are mutually exclusive.
      # Policy 1 (for specific pods)
      apiVersion: security.istio.io/v1beta1
      kind: PeerAuthentication
      metadata:
        name: policy-for-pods-a
        namespace: <your-workload-namespace>
      spec:
        selector:
          matchLabels:
            app: my-app
            tier: frontend
        mtls:
          mode: STRICT
      ---
      # Policy 2 (for other specific pods)
      apiVersion: security.istio.io/v1beta1
      kind: PeerAuthentication
      metadata:
        name: policy-for-pods-b
        namespace: <your-workload-namespace>
      spec:
        selector:
          matchLabels:
            app: my-app
            tier: backend
        mtls:
          mode: PERMISSIVE
      
    • Why it works: When multiple policies in the same namespace select the same workload, Istio’s behavior is undefined. By ensuring selectors are distinct and do not overlap for any given workload, you eliminate this ambiguity, allowing the intended policy to be applied.
  4. Global mesh Policy as the Intended Default:

    • Diagnosis: You expect a cluster-wide mesh policy to be active, but it’s not taking effect. This often happens if a namespace-scoped policy (even an empty one) exists and has higher precedence.
      kubectl get peerauthentication -n <your-workload-namespace>
      kubectl get peerauthentication -n istio-system
      kubectl get peerauthentication --all-namespaces | grep mesh
      
      Check if any PeerAuthentication exists in istio-system or your workload’s namespace.
    • Fix: If a namespace-scoped policy is present and you intend for the mesh policy to be the effective default, delete the namespace-scoped policy.
      kubectl delete peerauthentication <namespace-scoped-policy-name> -n <namespace>
      
      Ensure your mesh policy is correctly configured, for example, to enforce STRICT mTLS:
      apiVersion: security.istio.io/v1beta1
      kind: PeerAuthentication
      metadata:
        name: default
        namespace: istio-system
      spec:
        mtls:
          mode: STRICT
      
    • Why it works: Namespace-scoped policies (including those in istio-system) always take precedence over mesh (cluster-scoped) policies. Removing the interfering namespace-scoped policy allows the mesh policy to be the active one for workloads that are not covered by any other specific namespace-scoped policy.
  5. Incorrectly Applied mesh Policy (e.g., wrong name or namespace):

    • Diagnosis: You’ve created a PeerAuthentication policy with namespace: istio-system and name: default, but it’s not being enforced. Double-check the metadata.name and metadata.namespace fields. The only PeerAuthentication policy that applies cluster-wide to all workloads (unless overridden by a namespace-scoped policy) must be named default and reside in the istio-system namespace.
      kubectl get peerauthentication default -n istio-system -o yaml
      
    • Fix: Ensure your global PeerAuthentication policy is named default and is in the istio-system namespace. If it has a different name or is in another namespace, delete it and recreate it with the correct name and namespace.
      # Delete if it exists with the wrong name/namespace
      kubectl delete peerauthentication <wrong-name> -n <wrong-namespace>
      # Create the correct one
      cat <<EOF | kubectl apply -f -
      apiVersion: security.istio.io/v1beta1
      kind: PeerAuthentication
      metadata:
        name: default
        namespace: istio-system
      spec:
        mtls:
          mode: STRICT # Or PERMISSIVE, DISABLE
      EOF
      
    • Why it works: Istio specifically looks for a PeerAuthentication resource named default in the istio-system namespace to apply as the mesh-wide policy. Any other name or namespace will cause it to be treated as a regular namespace-scoped policy, subject to selector matching and precedence rules.
  6. Mismatched spec.selector Labels:

    • Diagnosis: You have a PeerAuthentication policy with a spec.selector defined, but the labels on your workload pods do not match the selector.
      # Check your PeerAuthentication policy
      kubectl get peerauthentication <policy-name> -n <namespace> -o yaml
      
      # Check the labels on your workload pods
      kubectl get pods -n <namespace> --show-labels
      
      Compare the spec.selector.matchLabels or spec.selector.matchExpressions with the output of kubectl get pods --show-labels.
    • Fix: Update either the policy’s selector or the workload’s labels to ensure they match.
      # Policy with specific selector
      apiVersion: security.istio.io/v1beta1
      kind: PeerAuthentication
      metadata:
        name: my-app-policy
        namespace: <your-workload-namespace>
      spec:
        selector:
          matchLabels:
            app: my-application # This label must exist on the pods
            version: v1
        mtls:
          mode: STRICT
      
      If your pods have labels app: my-application and version: v1, this policy will apply. If they have different labels, it won’t.
    • Why it works: A PeerAuthentication policy with a spec.selector only applies to workloads (pods) whose labels precisely match the selector. If there’s a mismatch, the policy is effectively ignored for that workload, and Istio falls back to the next applicable policy based on precedence.

After resolving these conflicts, the next error you might encounter is related to authorization policies (AuthorizationPolicy) not being able to enforce rules due to mTLS being in PERMISSIVE mode when STRICT was expected by the authorization policy.

Want structured learning?

Take the full Istio course →