Kustomize’s patchesStrategicMerge is a powerful tool for modifying existing Kubernetes manifests, but it can get surprisingly tricky when dealing with annotations and labels.
This is what happens when Kustomize, specifically during a patchesStrategicMerge operation, fails to correctly apply or ignore annotations and labels as you expect, leading to unexpected merge outcomes. The core issue is that Kustomize’s strategic merge logic, while powerful, treats annotations and labels as special fields that are always merged, even when you might want to replace or ignore them.
Here’s a breakdown of the common causes and how to fix them:
Cause 1: Default Merge Behavior for Annotations and Labels
Diagnosis:
When you use patchesStrategicMerge with a patch that targets annotations or labels, Kustomize, by default, merges the contents of the patch’s annotations/labels with the existing ones. It doesn’t replace them unless explicitly told to.
Fix:
To replace annotations or labels entirely, you need to provide a patch that explicitly lists the keys and values you want to be present in the final object. If you want to remove an annotation or label, you can’t do it directly with patchesStrategicMerge in a clean way; you’d typically use a different patching strategy or a separate tool. However, to replace them:
# base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
annotations:
existing-annotation: "true"
another-annotation: "value"
# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patchesStrategicMerge:
- deployment-patch.yaml
# overlays/production/deployment-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
annotations:
existing-annotation: "false" # This will ADD or REPLACE existing-annotation
new-annotation: "important" # This will ADD new-annotation
Running kustomize build overlays/production will result in:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
annotations:
existing-annotation: "false" # Replaced
another-annotation: "value" # Kept from base
new-annotation: "important" # Added
This works because Kustomize’s strategic merge for maps (like annotations and labels) is additive. If a key exists in both the base and the patch, the patch’s value wins. If it only exists in one, it’s kept.
Cause 2: Misunderstanding patchesJson6902 vs. patchesStrategicMerge
Diagnosis:
You might be trying to use patchesStrategicMerge expecting it to behave like a JSON patch (where you can specify "remove" operations), but it doesn’t. patchesJson6902 is the correct Kustomize directive for JSON patches.
Fix:
If you need to remove specific annotations or labels, use patchesJson6902.
# base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
annotations:
to-remove-annotation: "true"
keep-this-annotation: "value"
# overlays/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patchesJson6902:
- target:
kind: Deployment
name: my-app
patch: |-
- op: remove
path: /metadata/annotations/to-remove-annotation
Running kustomize build overlays/staging will result in:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
annotations:
keep-this-annotation: "value" # Removed
This works because JSON Patch operations (RFC 6902) explicitly support "remove" operations targeting specific paths within the JSON document, including nested maps like annotations and labels.
Cause 3: Overlapping Patches and Unexpected Merges
Diagnosis:
If you have multiple patchesStrategicMerge entries or a patchesStrategicMerge combined with a patchesJson6902 targeting the same resource, the order of application and the interaction between merge and remove operations can lead to confusion. Kustomize applies patchesStrategicMerge first, then patchesJson6902.
Fix:
Carefully review the sequence of operations. If you’re trying to remove an annotation that is also being added or modified by a strategic merge, the strategic merge might add it back before the JSON patch can remove it, or vice versa, depending on the exact structure. It’s often cleaner to consolidate changes within a single patch type if possible. For example, if you want to remove an annotation and add a new one, and the existing one needs a specific value, it’s usually best to use patchesStrategicMerge for the replacement and avoid trying to remove it first with patchesJson6902.
If you must remove an annotation that was present in the base and then add a new one, and the strategic merge isn’t behaving as expected, consider using patchesJson6902 for both the removal and the addition of new keys.
# overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patchesJson6902:
- target:
kind: Deployment
name: my-app
patch: |-
- op: remove
path: /metadata/annotations/to-remove-annotation
- op: add
path: /metadata/annotations/new-annotation
value: "added-via-jsonpatch"
This works because JSON Patch allows for multiple operations in a single patch document, which are applied atomically, giving you precise control over the final state of the annotations map.
Cause 4: Incorrect Target Specificity in Patches
Diagnosis: Your patch might be targeting the wrong resource or the wrong version of a resource if you have multiple resources with the same name but different kinds, or if you’re using generic selectors. This is less about annotations/labels themselves and more about Kustomize finding the right object to apply the annotation/label patch to.
Fix:
Ensure your target in patchesJson6902 or the name and kind in patchesStrategicMerge are precise. For patchesStrategicMerge, the name and kind are implicitly derived from the patch file itself. For patchesJson6902, they are explicit.
# overlays/testing/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patchesJson6902:
- target:
kind: Deployment
name: my-app # Make sure this is the exact name
group: apps # Specify group if needed, e.g., for CRDs
version: v1 # Specify version if needed
patch: |-
- op: replace
path: /metadata/annotations/version
value: "v2.1.0"
This works because Kustomize uses the kind, name, group, and version to uniquely identify the Kubernetes object within the generated manifest set that the patch should be applied to.
Cause 5: Annotations/Labels on Different Resource Types
Diagnosis:
You might be applying a patch intended for a Deployment’s annotations to a Service’s annotations, or vice-versa, if the patch file’s kind and name don’t precisely match.
Fix:
Double-check that the kind and metadata.name fields within your patch file (for patchesStrategicMerge) or the target section (for patchesJson6902) correctly identify the resource you intend to modify.
# overlays/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patchesStrategicMerge:
- deployment-annotations-patch.yaml
- service-annotations-patch.yaml
# overlays/staging/deployment-annotations-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
annotations:
deployment-specific: "true"
# overlays/staging/service-annotations-patch.yaml
apiVersion: v1
kind: Service
metadata:
name: my-app-service # Note: different name from deployment
annotations:
service-specific: "true"
This works because Kustomize matches patches to resources based on kind and name. If these don’t align, the patch won’t be applied to the intended resource, and no error will be raised unless the patch itself is malformed.
The next error you’ll likely encounter is a "resource already exists" error if you’ve accidentally created duplicate resources due to misconfigured patches or if your base configuration and patches are creating conflicting identical resources.