Helm upgrade failed because the Kubernetes API server rejected a change to a resource it considered immutable.
Common Causes and Fixes for Failed Helm Upgrades
-
Immutable Field Changes:
- Diagnosis: Helm will usually report an error like
admission webhook "..." denied the request: ... field is immutable. This means you tried to change a field that Kubernetes doesn’t allow to be modified after a resource is created. Common culprits includeselectorin Deployments,portsin Services, orstrategyin StatefulSets. - Cause: You’ve updated your Helm chart to modify a field that cannot be changed post-creation.
- Fix: You must delete the existing resource and let Helm recreate it. For example, if the
selectorin a Deployment is immutable:
Why it works: Deleting the resource removes the immutable field constraint, allowing Helm to create a new one with the updated configuration.# Find the specific resource kubectl get deployment <deployment-name> -n <namespace> -o yaml # Delete the resource (this will cause downtime) kubectl delete deployment <deployment-name> -n <namespace> # Then try the helm upgrade again helm upgrade <release-name> <chart-path> -n <namespace> - Cause: The
portssection of aServiceof typeLoadBalancerorNodePorthas been changed. - Fix: Similar to Deployments, you’ll need to delete and recreate the Service.
Why it works: The Service object is removed, and Helm creates a new one with the correct port configuration.kubectl delete service <service-name> -n <namespace> helm upgrade <release-name> <chart-path> -n <namespace> - Cause: Trying to change the
strategyin aStatefulSet. - Fix: Delete the StatefulSet.
Why it works: The StatefulSet’skubectl delete statefulset <statefulset-name> -n <namespace> helm upgrade <release-name> <chart-path> -n <namespace>strategycan only be set at creation time. Deleting it allows Helm to provision a new one with the desired strategy.
- Diagnosis: Helm will usually report an error like
-
Resource Quotas Exceeded:
- Diagnosis: Errors like
forbidden: exceeding quota: <quota-name>, requested: <resource-type> <count>, available: <remaining-count>. This means the namespace has aResourceQuotaobject that prevents you from creating more of a certain resource type (e.g., Pods, Services, PVCs). - Cause: Your Helm chart is trying to create new resources, but the namespace’s quota has been hit for that resource type.
- Fix: Increase the quota in the namespace or reduce the number of resources your Helm chart is deploying. To view quotas:
To edit a quota (e.g., increasekubectl get resourcequotas -n <namespace>podslimit):
Changekubectl edit resourcequota <quota-name> -n <namespace>spec.hard.podsto a higher value. Why it works: TheResourceQuotaobject enforces limits. By increasing the limit, you allow the new resources to be created.
- Diagnosis: Errors like
-
Admission Webhook Failures (Non-Immutable):
- Diagnosis: Generic errors from an admission webhook, often starting with
admission webhook "..." denied the request:. Unlike immutable field errors, these are usually due to custom validation rules defined in the webhook. The message will specify the exact violation. - Cause: A custom admission controller (often part of a service mesh like Istio or a security policy manager like Kyverno/OPA Gatekeeper) is rejecting the resource based on its configuration.
- Fix: Review the specific rejection message from the webhook. It will tell you what rule was violated. For example, if a policy prevents
hostPathvolumes:
You would then need to modify your Helm chart to remove the offending configuration (e.g., remove the# Example policy violation message admission webhook "validate.kyverno.svc-1" denied the request: container-security-context: | - policy: disallow-hostpath rule: hostPath is disallowedhostPathvolume mount from your Pod template). Why it works: Admission webhooks act as gatekeepers, enforcing policies. Removing the configuration that violates the policy allows the resource to be admitted.
- Diagnosis: Generic errors from an admission webhook, often starting with
-
RBAC Permissions Issues:
- Diagnosis: Errors like
Error: UPGRADE FAILED: failed to create resource: ... User "system:serviceaccount:<namespace>:<release-name>-controller" cannot create resource "deployments" in API group "" in the namespace "<namespace>"orforbidden: User "..." cannot get .... - Cause: The Service Account that Helm uses (or the user running
helm upgradeif not using Tiller or Helm 3’s direct API interaction) lacks the necessary Role-Based Access Control (RBAC) permissions to create, update, or delete the resources defined in your chart. - Fix: Grant the required permissions by creating or updating
RoleorClusterRoleandRoleBindingorClusterRoleBindingobjects. For example, to allow a service account to manage Deployments:
Apply this manifest:apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: <namespace> name: deployment-manager rules: - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: deploy-to-deployment-manager namespace: <namespace> subjects: - kind: ServiceAccount name: <helm-service-account-name> # e.g., my-release-controller namespace: <namespace> roleRef: kind: Role name: deployment-manager apiGroup: rbac.authorization.k8s.iokubectl apply -f rbac.yaml. Why it works: RBAC rules define what actions a subject (like a Service Account) can perform on specific resources. Granting the correct permissions allows Helm to interact with the Kubernetes API as needed.
- Diagnosis: Errors like
-
StorageClass Not Found or Incorrect:
- Diagnosis: Errors related to PersistentVolumeClaims (PVCs), such as
failed to provision volume with StorageClass "..." : no such StorageClass: <storage-class-name>orwaiting for first consumer to be created before binding. - Cause: Your chart is configured to use a
StorageClassthat either doesn’t exist in the cluster or has an incorrect name specified in the PVC template. - Fix: Ensure the
StorageClassexists and is correctly named.
If thekubectl get storageclassStorageClassis missing, you’ll need to create it or use an existing one. If the name is wrong in your chart’svalues.yamlor PVC template, correct it. For example, if yourvalues.yamlhasstorageClassName: "my-ssd-storage"but the actual class isssd-storage:
Then run# In values.yaml persistence: storageClassName: "ssd-storage" # Corrected namehelm upgrade. Why it works: Kubernetes usesStorageClassto dynamically provision PersistentVolumes. If the specified class is unknown, provisioning fails. Correcting the name allows the dynamic provisioning to succeed.
- Diagnosis: Errors related to PersistentVolumeClaims (PVCs), such as
-
CRD Version Mismatches or Missing:
- Diagnosis: Errors like
no matches for kind "MyCustomResource" in version "my.domain.com/v1"orfailed to create resource: ... the server could not find the requested resource. - Cause: Your Helm chart depends on Custom Resource Definitions (CRDs) that are either not installed in the cluster, or the version of the CRD specified in your chart is not compatible with what’s installed. Helm upgrades often don’t manage CRDs by default.
- Fix: Install or upgrade the CRDs manually before running
helm upgrade. Check your chart’s dependencies or the CRD’s documentation for the correct installation method. Often, CRDs are provided as separate YAML files.
After applying the CRDs, run# Example: If CRDs are in a crds/ directory kubectl apply -f crds/ # Or apply specific CRD files kubectl apply -f https://raw.githubusercontent.com/some/repo/main/crds/mycrd.yamlhelm upgrade. Why it works: CRDs define custom Kubernetes objects. Without the correct CRD installed, the Kubernetes API server doesn’t recognize the custom resource types your chart is trying to create.
- Diagnosis: Errors like
The next error you’ll likely encounter after fixing these is a pending state on a Pod due to insufficient CPU/memory resources in the node, or a CrashLoopBackOff if the application itself has a bug.