Kustomize’s immutable ConfigMaps will prevent your Deployments from updating.

The Problem

When Kustomize generates a ConfigMap and it’s mounted by a Deployment, Kubernetes’s immutable ConfigMap feature prevents the Deployment from updating if the ConfigMap changes. This happens because Kubernetes considers the ConfigMap to be immutable after creation, and any modification to it will cause a validation error. You’ll see unchanged statuses for your pods, and your application won’t pick up the new configuration.

Common Causes and Fixes

  1. The Immutable ConfigMap Feature is Enabled (Default Behavior):

    • Diagnosis: Create a ConfigMap with immutable: true in its data field. Then, try to update a value in that ConfigMap.
      kubectl apply -f - <<EOF
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: my-immutable-config
      data:
        key: initial-value
      immutable: true
      EOF
      
      # Now try to update it
      kubectl apply -f - <<EOF
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: my-immutable-config
      data:
        key: new-value
      immutable: true
      EOF
      
      You’ll get an error like: admission webhook "gatekeeper.sh" denied the request: [immutable field is immutable].
    • Fix: Remove immutable: true from the ConfigMap definition. Kustomize does not set this by default unless you explicitly add it in your kustomization.yaml. If you did add it, remove that line. The default behavior is mutable, which is what you want for rolling updates.
    • Why it works: By removing immutable: true, you allow Kubernetes to treat the ConfigMap as mutable, meaning its contents can be updated in place.
  2. Kustomize Is Not Actually Generating a New ConfigMap:

    • Diagnosis: When you run kustomize build ., examine the output. If your ConfigMap definition hasn’t changed at all between builds, Kustomize won’t generate a new object. Check the checksum annotation that Kustomize adds to ConfigMaps and Secrets.
      kustomize build . > manifest.yaml
      grep my-config manifest.yaml
      
      Look for a line like metadata.annotations."kustomize.config.k8s.io/checksum". If this checksum isn’t changing, Kustomize isn’t seeing a difference.
    • Fix: Ensure that the data or files section in your kustomization.yaml that defines the ConfigMap is actually changing. This usually means changing a value directly, or if you’re using files, changing the content of the file being included.
      # kustomization.yaml
      apiVersion: kustomize.config.k8s.io/v1beta1
      kind: Kustomization
      resources:
      - deployment.yaml
      
      configMapGenerator:
      - name: my-app-config
        literals:
        - app.properties="some.setting=new_value" # Change this value
      
      Alternatively, if using files:
      # kustomization.yaml
      apiVersion: kustomize.config.k8s.io/v1beta1
      kind: Kustomization
      resources:
      - deployment.yaml
      
      configMapGenerator:
      - name: my-app-config
        files:
        - app.properties # Ensure app.properties file content has changed
      
    • Why it works: Kustomize’s configMapGenerator (and secretGenerator) use a checksum of the generated content to determine if a new ConfigMap object needs to be created. If the content changes, Kustomize generates a new ConfigMap with a new name (including a hash of the content), and the Deployment’s spec.template.spec.volumes will reference this new ConfigMap.
  3. Deployment’s Pod Template Spec Is Not Set to Update:

    • Diagnosis: Even if Kustomize generates a new ConfigMap, the Deployment won’t restart if its spec.template.spec.volumes section isn’t correctly referencing the new ConfigMap generated by Kustomize. The spec.template is what gets hashed for rolling updates.
      # Example deployment snippet
      spec:
        template:
          metadata:
            labels:
              app: my-app
          spec:
            containers:
            - name: my-app-container
              image: my-image:latest
              volumeMounts:
              - name: config-volume
                mountPath: /etc/config
            volumes:
            - name: config-volume
              configMap:
                name: my-app-config # This MUST match the name Kustomize generates
      
    • Fix: Ensure your deployment.yaml correctly references the ConfigMap name generated by Kustomize. Kustomize’s configMapGenerator automatically names the ConfigMap like my-app-config-abcdef1234. Your Deployment’s volumes section must point to this specific, generated name. A common mistake is hardcoding the base name (my-app-config) instead of letting Kustomize manage the full, hashed name.
      # kustomization.yaml
      apiVersion: kustomize.config.k8s.io/v1beta1
      kind: Kustomization
      resources:
      - deployment-template.yaml # A template that references a ConfigMap name
      
      configMapGenerator:
      - name: my-app-config # Kustomize will append a hash here
        literals:
        - app.properties="some.setting=new_value"
      
      patches:
      - target:
          kind: Deployment
          name: my-deployment # Assuming your deployment is named my-deployment
        patch: |-
          apiVersion: apps/v1
          kind: Deployment
          spec:
            template:
              spec:
                volumes:
                - name: config-volume # Must match volume name in deployment-template.yaml
                  configMap:
                    name: my-app-config # This refers to the *base name* Kustomize will hash
      
      Kustomize will then substitute the actual generated ConfigMap name (e.g., my-app-config-abcdef1234) into the Deployment’s spec.template.spec.volumes.configMap.name field.
    • Why it works: Kubernetes calculates a hash of the spec.template section of a Deployment. If any part of spec.template changes, including the referenced ConfigMap name, the hash changes. This change triggers a rolling update because Kubernetes sees the spec.template as a new version.
  4. Missing kustomize.config.k8s.io/v1beta1 API Version in kustomization.yaml:

    • Diagnosis: If your kustomization.yaml doesn’t start with the correct API version, Kustomize might not process generators or patches correctly.
      # Incorrect kustomization.yaml start
      kind: Kustomization
      resources:
      - deployment.yaml
      
    • Fix: Ensure your kustomization.yaml has the correct API version at the top:
      # Correct kustomization.yaml start
      apiVersion: kustomize.config.k8s.io/v1beta1
      kind: Kustomization
      resources:
      - deployment.yaml
      
    • Why it works: This API version tells the Kustomize controller or CLI that it’s processing a Kustomization file and should enable all its features, including generators and patching, which are essential for managing ConfigMap updates.
  5. Using kubectl edit or kubectl apply Directly on ConfigMaps:

    • Diagnosis: If you manually edit a ConfigMap that was originally managed by Kustomize using kubectl edit configmap <configmap-name>, you are bypassing Kustomize’s generation process. The next time Kustomize builds and applies, it might generate a new ConfigMap, leaving the manually edited one stale and the Deployment still pointing to the old one.
    • Fix: All changes to configurations managed by Kustomize should be made within your Kustomize source files (kustomization.yaml, overlaid files, etc.). After updating your Kustomize source, run kustomize build . | kubectl apply -f - or use your CI/CD pipeline to apply the updated manifests. Never edit Kustomize-managed resources directly with kubectl edit.
    • Why it works: This ensures that your Git repository remains the single source of truth for your application’s configuration. Kustomize regenerates the ConfigMap with a new hash, and the Deployment template is updated to point to this new ConfigMap, triggering the restart.
  6. ConfigMap is Not Mounted by the Deployment:

    • Diagnosis: The ConfigMap might be generated and updated, but if no pods are actually mounting it, there’s no dependency to break. Check the volumeMounts and volumes sections of your Deployment’s pod template.
    • Fix: Add a volumeMount to your container spec and a corresponding volume definition in the pod spec that references the ConfigMap.
      # In your deployment.yaml (or template used by Kustomize)
      spec:
        template:
          spec:
            containers:
            - name: my-app-container
              # ... other container settings
              volumeMounts:
              - name: config-volume
                mountPath: /app/config
            volumes:
            - name: config-volume
              configMap:
                name: my-app-config # Base name, Kustomize adds hash
      
    • Why it works: This establishes the link between the Pod and the ConfigMap. When the ConfigMap referenced by the volumes section changes (due to Kustomize generating a new version), Kubernetes detects this change and orchestrates a rollout of the Deployment to update the Pods.

After fixing these issues, the next error you’ll likely encounter is a CrashLoopBackOff if your application fails to start due to an invalid configuration value within the new ConfigMap.

Want structured learning?

Take the full Kustomize course →