Kustomize’s immutable ConfigMaps will prevent your Deployments from updating.
The Problem
When Kustomize generates a ConfigMap and it’s mounted by a Deployment, Kubernetes’s immutable ConfigMap feature prevents the Deployment from updating if the ConfigMap changes. This happens because Kubernetes considers the ConfigMap to be immutable after creation, and any modification to it will cause a validation error. You’ll see unchanged statuses for your pods, and your application won’t pick up the new configuration.
Common Causes and Fixes
-
The Immutable ConfigMap Feature is Enabled (Default Behavior):
- Diagnosis: Create a ConfigMap with
immutable: truein itsdatafield. Then, try to update a value in that ConfigMap.
You’ll get an error like:kubectl apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: my-immutable-config data: key: initial-value immutable: true EOF # Now try to update it kubectl apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: my-immutable-config data: key: new-value immutable: true EOFadmission webhook "gatekeeper.sh" denied the request: [immutable field is immutable]. - Fix: Remove
immutable: truefrom the ConfigMap definition. Kustomize does not set this by default unless you explicitly add it in yourkustomization.yaml. If you did add it, remove that line. The default behavior is mutable, which is what you want for rolling updates. - Why it works: By removing
immutable: true, you allow Kubernetes to treat the ConfigMap as mutable, meaning its contents can be updated in place.
- Diagnosis: Create a ConfigMap with
-
Kustomize Is Not Actually Generating a New ConfigMap:
- Diagnosis: When you run
kustomize build ., examine the output. If your ConfigMap definition hasn’t changed at all between builds, Kustomize won’t generate a new object. Check thechecksumannotation that Kustomize adds to ConfigMaps and Secrets.
Look for a line likekustomize build . > manifest.yaml grep my-config manifest.yamlmetadata.annotations."kustomize.config.k8s.io/checksum". If this checksum isn’t changing, Kustomize isn’t seeing a difference. - Fix: Ensure that the
dataorfilessection in yourkustomization.yamlthat defines the ConfigMap is actually changing. This usually means changing a value directly, or if you’re usingfiles, changing the content of the file being included.
Alternatively, if using# kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml configMapGenerator: - name: my-app-config literals: - app.properties="some.setting=new_value" # Change this valuefiles:# kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml configMapGenerator: - name: my-app-config files: - app.properties # Ensure app.properties file content has changed - Why it works: Kustomize’s
configMapGenerator(andsecretGenerator) use a checksum of the generated content to determine if a new ConfigMap object needs to be created. If the content changes, Kustomize generates a new ConfigMap with a new name (including a hash of the content), and the Deployment’sspec.template.spec.volumeswill reference this new ConfigMap.
- Diagnosis: When you run
-
Deployment’s Pod Template Spec Is Not Set to Update:
- Diagnosis: Even if Kustomize generates a new ConfigMap, the Deployment won’t restart if its
spec.template.spec.volumessection isn’t correctly referencing the new ConfigMap generated by Kustomize. Thespec.templateis what gets hashed for rolling updates.# Example deployment snippet spec: template: metadata: labels: app: my-app spec: containers: - name: my-app-container image: my-image:latest volumeMounts: - name: config-volume mountPath: /etc/config volumes: - name: config-volume configMap: name: my-app-config # This MUST match the name Kustomize generates - Fix: Ensure your
deployment.yamlcorrectly references the ConfigMap name generated by Kustomize. Kustomize’sconfigMapGeneratorautomatically names the ConfigMap likemy-app-config-abcdef1234. Your Deployment’svolumessection must point to this specific, generated name. A common mistake is hardcoding the base name (my-app-config) instead of letting Kustomize manage the full, hashed name.
Kustomize will then substitute the actual generated ConfigMap name (e.g.,# kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment-template.yaml # A template that references a ConfigMap name configMapGenerator: - name: my-app-config # Kustomize will append a hash here literals: - app.properties="some.setting=new_value" patches: - target: kind: Deployment name: my-deployment # Assuming your deployment is named my-deployment patch: |- apiVersion: apps/v1 kind: Deployment spec: template: spec: volumes: - name: config-volume # Must match volume name in deployment-template.yaml configMap: name: my-app-config # This refers to the *base name* Kustomize will hashmy-app-config-abcdef1234) into the Deployment’sspec.template.spec.volumes.configMap.namefield. - Why it works: Kubernetes calculates a hash of the
spec.templatesection of a Deployment. If any part ofspec.templatechanges, including the referenced ConfigMap name, the hash changes. This change triggers a rolling update because Kubernetes sees thespec.templateas a new version.
- Diagnosis: Even if Kustomize generates a new ConfigMap, the Deployment won’t restart if its
-
Missing
kustomize.config.k8s.io/v1beta1API Version inkustomization.yaml:- Diagnosis: If your
kustomization.yamldoesn’t start with the correct API version, Kustomize might not process generators or patches correctly.# Incorrect kustomization.yaml start kind: Kustomization resources: - deployment.yaml - Fix: Ensure your
kustomization.yamlhas the correct API version at the top:# Correct kustomization.yaml start apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - Why it works: This API version tells the Kustomize controller or CLI that it’s processing a Kustomization file and should enable all its features, including generators and patching, which are essential for managing ConfigMap updates.
- Diagnosis: If your
-
Using
kubectl editorkubectl applyDirectly on ConfigMaps:- Diagnosis: If you manually edit a ConfigMap that was originally managed by Kustomize using
kubectl edit configmap <configmap-name>, you are bypassing Kustomize’s generation process. The next time Kustomize builds and applies, it might generate a new ConfigMap, leaving the manually edited one stale and the Deployment still pointing to the old one. - Fix: All changes to configurations managed by Kustomize should be made within your Kustomize source files (
kustomization.yaml, overlaid files, etc.). After updating your Kustomize source, runkustomize build . | kubectl apply -f -or use your CI/CD pipeline to apply the updated manifests. Never edit Kustomize-managed resources directly withkubectl edit. - Why it works: This ensures that your Git repository remains the single source of truth for your application’s configuration. Kustomize regenerates the ConfigMap with a new hash, and the Deployment template is updated to point to this new ConfigMap, triggering the restart.
- Diagnosis: If you manually edit a ConfigMap that was originally managed by Kustomize using
-
ConfigMap is Not Mounted by the Deployment:
- Diagnosis: The ConfigMap might be generated and updated, but if no pods are actually mounting it, there’s no dependency to break. Check the
volumeMountsandvolumessections of your Deployment’s pod template. - Fix: Add a
volumeMountto your container spec and a correspondingvolumedefinition in the pod spec that references the ConfigMap.# In your deployment.yaml (or template used by Kustomize) spec: template: spec: containers: - name: my-app-container # ... other container settings volumeMounts: - name: config-volume mountPath: /app/config volumes: - name: config-volume configMap: name: my-app-config # Base name, Kustomize adds hash - Why it works: This establishes the link between the Pod and the ConfigMap. When the ConfigMap referenced by the
volumessection changes (due to Kustomize generating a new version), Kubernetes detects this change and orchestrates a rollout of the Deployment to update the Pods.
- Diagnosis: The ConfigMap might be generated and updated, but if no pods are actually mounting it, there’s no dependency to break. Check the
After fixing these issues, the next error you’ll likely encounter is a CrashLoopBackOff if your application fails to start due to an invalid configuration value within the new ConfigMap.