Helm upgrade failed because the Kubernetes API server rejected a change to a resource it considered immutable.

Common Causes and Fixes for Failed Helm Upgrades

  1. Immutable Field Changes:

    • Diagnosis: Helm will usually report an error like admission webhook "..." denied the request: ... field is immutable. This means you tried to change a field that Kubernetes doesn’t allow to be modified after a resource is created. Common culprits include selector in Deployments, ports in Services, or strategy in StatefulSets.
    • Cause: You’ve updated your Helm chart to modify a field that cannot be changed post-creation.
    • Fix: You must delete the existing resource and let Helm recreate it. For example, if the selector in a Deployment is immutable:
      # Find the specific resource
      kubectl get deployment <deployment-name> -n <namespace> -o yaml
      
      # Delete the resource (this will cause downtime)
      kubectl delete deployment <deployment-name> -n <namespace>
      
      # Then try the helm upgrade again
      helm upgrade <release-name> <chart-path> -n <namespace>
      
      Why it works: Deleting the resource removes the immutable field constraint, allowing Helm to create a new one with the updated configuration.
    • Cause: The ports section of a Service of type LoadBalancer or NodePort has been changed.
    • Fix: Similar to Deployments, you’ll need to delete and recreate the Service.
      kubectl delete service <service-name> -n <namespace>
      helm upgrade <release-name> <chart-path> -n <namespace>
      
      Why it works: The Service object is removed, and Helm creates a new one with the correct port configuration.
    • Cause: Trying to change the strategy in a StatefulSet.
    • Fix: Delete the StatefulSet.
      kubectl delete statefulset <statefulset-name> -n <namespace>
      helm upgrade <release-name> <chart-path> -n <namespace>
      
      Why it works: The StatefulSet’s strategy can only be set at creation time. Deleting it allows Helm to provision a new one with the desired strategy.
  2. Resource Quotas Exceeded:

    • Diagnosis: Errors like forbidden: exceeding quota: <quota-name>, requested: <resource-type> <count>, available: <remaining-count>. This means the namespace has a ResourceQuota object that prevents you from creating more of a certain resource type (e.g., Pods, Services, PVCs).
    • Cause: Your Helm chart is trying to create new resources, but the namespace’s quota has been hit for that resource type.
    • Fix: Increase the quota in the namespace or reduce the number of resources your Helm chart is deploying. To view quotas:
      kubectl get resourcequotas -n <namespace>
      
      To edit a quota (e.g., increase pods limit):
      kubectl edit resourcequota <quota-name> -n <namespace>
      
      Change spec.hard.pods to a higher value. Why it works: The ResourceQuota object enforces limits. By increasing the limit, you allow the new resources to be created.
  3. Admission Webhook Failures (Non-Immutable):

    • Diagnosis: Generic errors from an admission webhook, often starting with admission webhook "..." denied the request:. Unlike immutable field errors, these are usually due to custom validation rules defined in the webhook. The message will specify the exact violation.
    • Cause: A custom admission controller (often part of a service mesh like Istio or a security policy manager like Kyverno/OPA Gatekeeper) is rejecting the resource based on its configuration.
    • Fix: Review the specific rejection message from the webhook. It will tell you what rule was violated. For example, if a policy prevents hostPath volumes:
      # Example policy violation message
      admission webhook "validate.kyverno.svc-1" denied the request:
      container-security-context: |
        - policy: disallow-hostpath
          rule: hostPath is disallowed
      
      You would then need to modify your Helm chart to remove the offending configuration (e.g., remove the hostPath volume mount from your Pod template). Why it works: Admission webhooks act as gatekeepers, enforcing policies. Removing the configuration that violates the policy allows the resource to be admitted.
  4. RBAC Permissions Issues:

    • Diagnosis: Errors like Error: UPGRADE FAILED: failed to create resource: ... User "system:serviceaccount:<namespace>:<release-name>-controller" cannot create resource "deployments" in API group "" in the namespace "<namespace>" or forbidden: User "..." cannot get ....
    • Cause: The Service Account that Helm uses (or the user running helm upgrade if not using Tiller or Helm 3’s direct API interaction) lacks the necessary Role-Based Access Control (RBAC) permissions to create, update, or delete the resources defined in your chart.
    • Fix: Grant the required permissions by creating or updating Role or ClusterRole and RoleBinding or ClusterRoleBinding objects. For example, to allow a service account to manage Deployments:
      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        namespace: <namespace>
        name: deployment-manager
      rules:
      - apiGroups: ["apps"]
        resources: ["deployments"]
        verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: deploy-to-deployment-manager
        namespace: <namespace>
      subjects:
      - kind: ServiceAccount
        name: <helm-service-account-name> # e.g., my-release-controller
        namespace: <namespace>
      roleRef:
        kind: Role
        name: deployment-manager
        apiGroup: rbac.authorization.k8s.io
      
      Apply this manifest: kubectl apply -f rbac.yaml. Why it works: RBAC rules define what actions a subject (like a Service Account) can perform on specific resources. Granting the correct permissions allows Helm to interact with the Kubernetes API as needed.
  5. StorageClass Not Found or Incorrect:

    • Diagnosis: Errors related to PersistentVolumeClaims (PVCs), such as failed to provision volume with StorageClass "..." : no such StorageClass: <storage-class-name> or waiting for first consumer to be created before binding.
    • Cause: Your chart is configured to use a StorageClass that either doesn’t exist in the cluster or has an incorrect name specified in the PVC template.
    • Fix: Ensure the StorageClass exists and is correctly named.
      kubectl get storageclass
      
      If the StorageClass is missing, you’ll need to create it or use an existing one. If the name is wrong in your chart’s values.yaml or PVC template, correct it. For example, if your values.yaml has storageClassName: "my-ssd-storage" but the actual class is ssd-storage:
      # In values.yaml
      persistence:
        storageClassName: "ssd-storage" # Corrected name
      
      Then run helm upgrade. Why it works: Kubernetes uses StorageClass to dynamically provision PersistentVolumes. If the specified class is unknown, provisioning fails. Correcting the name allows the dynamic provisioning to succeed.
  6. CRD Version Mismatches or Missing:

    • Diagnosis: Errors like no matches for kind "MyCustomResource" in version "my.domain.com/v1" or failed to create resource: ... the server could not find the requested resource.
    • Cause: Your Helm chart depends on Custom Resource Definitions (CRDs) that are either not installed in the cluster, or the version of the CRD specified in your chart is not compatible with what’s installed. Helm upgrades often don’t manage CRDs by default.
    • Fix: Install or upgrade the CRDs manually before running helm upgrade. Check your chart’s dependencies or the CRD’s documentation for the correct installation method. Often, CRDs are provided as separate YAML files.
      # Example: If CRDs are in a crds/ directory
      kubectl apply -f crds/
      # Or apply specific CRD files
      kubectl apply -f https://raw.githubusercontent.com/some/repo/main/crds/mycrd.yaml
      
      After applying the CRDs, run helm upgrade. Why it works: CRDs define custom Kubernetes objects. Without the correct CRD installed, the Kubernetes API server doesn’t recognize the custom resource types your chart is trying to create.

The next error you’ll likely encounter after fixing these is a pending state on a Pod due to insufficient CPU/memory resources in the node, or a CrashLoopBackOff if the application itself has a bug.

Want structured learning?

Take the full Helm course →