The kube-controller-manager gave up on telling the kubelet to remove resources within a namespace, because a finalizer wasn’t being removed.
This usually happens because one or more resources inside the namespace have a finalizer that is stuck, preventing the namespace from being fully deleted. The kube-controller-manager tries to delete the namespace, sees the finalizer, and asks the controller responsible for that resource to remove it. If that controller never acknowledges the removal, the namespace remains in Terminating state.
Here are the most common reasons and how to fix them:
1. Stuck kubernetes.io/pv-protection Finalizer on a PersistentVolume
This is by far the most frequent culprit. If a PersistentVolume (PV) was bound to a PersistentVolumeClaim (PVC) within the namespace that’s now terminating, Kubernetes protects the PV from accidental deletion. If the PVC is gone but the PV still has this finalizer, the namespace can’t proceed.
Diagnosis:
Check if any PVs are still referencing the namespace or if a PV has the kubernetes.io/pv-protection finalizer.
kubectl get pv --all-namespaces -o jsonpath='{.items[?(@.spec.claimRef.namespace=="<NAMESPACE_NAME>")]}'
kubectl get pv -o jsonpath='{.items[?(@.metadata.deletionTimestamp!=null && @.metadata.finalizers[*]=="kubernetes.io/pv-protection")]}'
Replace <NAMESPACE_NAME> with the name of your stuck namespace. If the second command returns anything, you’ve found your problem.
Fix:
Manually remove the pv-protection finalizer from the problematic PV.
kubectl patch pv <PV_NAME> -p '{"metadata": {"finalizers": null}}'
Replace <PV_NAME> with the name of the PV identified in the diagnosis. This forces Kubernetes to remove the finalizer and allows the PV (and subsequently the namespace) to be deleted.
Why it works: The pv-protection finalizer is meant to prevent data loss by keeping the PV around if its PVC is deleted. By setting finalizers: null, you’re telling Kubernetes to ignore this protection and proceed with the PV’s deletion, which in turn allows the namespace’s deletion to complete.
2. Stuck Custom Resource Finalizers
Many operators and custom controllers add their own finalizers to custom resources (CRs) to manage their lifecycle. If the controller responsible for a CR within the terminating namespace crashes or is deleted improperly, its finalizers can get stuck.
Diagnosis: List all resources in the stuck namespace and look for any that have finalizers.
kubectl get all --namespace <NAMESPACE_NAME> -o json | jq '.items[] | select(.metadata.finalizers)'
Replace <NAMESPACE_NAME> with your stuck namespace. If this command returns any resources, note their kind and metadata.name. Then, check the status of the controller that manages that kind.
Fix: The ideal fix is to ensure the controller responsible for the CR is running and can process the deletion request. If the controller is permanently gone, you’ll need to manually remove the finalizer from the CR.
kubectl patch <CR_KIND> <CR_NAME> --namespace <NAMESPACE_NAME> -p '{"metadata": {"finalizers": null}}'
Replace <CR_KIND>, <CR_NAME>, and <NAMESPACE_NAME> with the details of the stuck CR.
Why it works: Similar to PV protection, custom finalizers signal to Kubernetes that a resource requires special cleanup before it can be deleted. Removing the finalizer bypasses this cleanup, allowing the resource (and thus the namespace) to be deleted immediately. This assumes you’ve already handled any necessary manual cleanup or that the resource is no longer needed.
3. Stuck kubernetes.io/pvc-protection Finalizer on a PersistentVolumeClaim
While less common than PV protection, a PVC can also have a protection finalizer, especially if it’s associated with an underlying storage provisioner that has its own cleanup logic.
Diagnosis:
Check for PVCs in the stuck namespace that have the pvc-protection finalizer.
kubectl get pvc --namespace <NAMESPACE_NAME> -o json | jq '.items[] | select(.metadata.deletionTimestamp!=null && .metadata.finalizers[]=="kubernetes.io/pvc-protection")'
Replace <NAMESPACE_NAME> with your stuck namespace.
Fix:
Remove the pvc-protection finalizer from the stuck PVC.
kubectl patch pvc <PVC_NAME> --namespace <NAMESPACE_NAME> -p '{"metadata": {"finalizers": null}}'
Replace <PVC_NAME> and <NAMESPACE_NAME> accordingly.
Why it works: This allows the PVC to be deleted, which can then trigger the deletion of its associated PV (if it was still bound) or simply allow the namespace deletion process to continue.
4. NetworkPolicy or other CNI-related Resources
Sometimes, network plugins or CNI (Container Network Interface) components might introduce their own finalizers or create resources that get stuck.
Diagnosis: Inspect all resources in the namespace, paying close attention to any CRDs or resources managed by your CNI plugin.
kubectl get all --namespace <NAMESPACE_NAME> -o wide
kubectl get crds --all-namespaces -o jsonpath='{.items[*].metadata.name}' | xargs -I {} kubectl get {} --namespace <NAMESPACE_NAME>
Look for resources with names or kinds that suggest a network component.
Fix: If you identify a network-related resource stuck with a finalizer, you’ll need to consult the documentation for your specific CNI plugin to understand its deletion process. Often, it involves manually patching the resource to remove the finalizer, similar to custom resources.
Why it works: The CNI plugin’s internal logic for cleaning up network configurations might be preventing the resource from being removed. Manually clearing the finalizer forces its deletion.
5. Stuck cloudprovider.kubernetes.io/lbs Finalizer on Service
If you have a Service of type LoadBalancer in your namespace, and the cloud provider integration is having trouble deleting the external load balancer, this finalizer can get stuck.
Diagnosis:
Check services in the namespace for the cloudprovider.kubernetes.io/lbs finalizer.
kubectl get service --namespace <NAMESPACE_NAME> -o json | jq '.items[] | select(.metadata.deletionTimestamp!=null && .metadata.finalizers[]=="cloudprovider.kubernetes.io/lbs")'
Replace <NAMESPACE_NAME> with your stuck namespace.
Fix: You’ll likely need to manually delete the cloud provider’s load balancer resource (e.g., an ELB in AWS, a LoadBalancer in GCP) and then remove the finalizer from the Kubernetes Service.
# Example for AWS:
aws elb delete-load-balancer --load-balancer-name <LB_NAME>
# Then patch the service:
kubectl patch service <SERVICE_NAME> --namespace <NAMESPACE_NAME> -p '{"metadata": {"finalizers": null}}'
Replace <LB_NAME> and <SERVICE_NAME> with your specific values.
Why it works: This finalizer is a hook for the cloud controller manager to ensure the cloud resource (the load balancer) is cleaned up before the Kubernetes Service object is deleted. If the cloud resource deletion fails, the finalizer persists. Manually cleaning the cloud resource and then removing the finalizer allows Kubernetes to complete the deletion.
6. etcd Issues or Controller Manager Crashes
In rare cases, the problem might not be a specific resource but a transient issue with etcd or the kube-controller-manager itself.
Diagnosis:
Check the logs of the kube-controller-manager pods.
kubectl logs -n kube-system <kube-controller-manager-pod-name>
Look for errors related to namespace deletion or finalizer processing. Also, check etcd health.
Fix:
If kube-controller-manager is crashing, restart it. If etcd is unhealthy, that’s a more serious cluster-wide issue that needs immediate attention. For transient etcd issues, a kube-controller-manager restart might resolve it.
Why it works: A healthy kube-controller-manager is essential for garbage collection and resource finalization. Restoring its health or resolving underlying etcd problems allows it to correctly process deletion requests.
After resolving the stuck finalizer, you should be able to delete the namespace. If you encounter issues with a Pod stuck in Terminating state, that’s the next common problem you’ll face.