A pod is failing in Minikube because the kubelet, the node agent responsible for running containers, has declared it unhealthy and stopped it.
Common Causes and Fixes for Failing Minikube Pods
1. Image Pull Error
- Diagnosis: Check the pod’s events.
Look forkubectl describe pod <pod-name> -n <namespace>ErrImagePullorImagePullBackOffin theEventssection. - Cause: Minikube couldn’t download the container image. This is often due to a typo in the image name/tag, the image not existing in the specified registry, or network issues preventing access to the registry.
- Fix:
- Typo/Non-existent Image: Correct the
imagefield in your Pod or Deployment YAML. For example, if you meantnginx:latest, ensure it’s spelled exactly like that. - Private Registry: If using a private registry, ensure you have created and applied a
docker-registrysecret and referenced it in your Pod spec usingimagePullSecrets.
Create the secret:apiVersion: v1 kind: Pod metadata: name: my-private-pod spec: containers: - name: my-app image: my-registry.com/my-app:v1.0 imagePullSecrets: - name: my-registry-secretkubectl create secret docker-registry my-registry-secret \ --docker-server=<your-registry-server> \ --docker-username=<your-username> \ --docker-password=<your-password> \ --docker-email=<your-email> - Network Issues: Verify Minikube can reach the registry. Try pulling the image manually within the Minikube VM if possible (though this is advanced). A simpler check is to ensure your host machine has internet access and no VPN or firewall is blocking Minikube’s access.
- Typo/Non-existent Image: Correct the
- Why it works: Kubernetes needs to fetch the container image to start a pod. If it can’t find or access the image, it will fail to start and report an error. Correcting the image reference or providing credentials/network access resolves this.
2. Liveness/Readiness Probe Failure
- Diagnosis: Again,
kubectl describe pod <pod-name> -n <namespace>. Look for events indicating probe failures, such asLiveness probe failedorReadiness probe failed. - Cause: The pod’s application isn’t responding to health checks as configured. This could be because the application crashed, is slow to start, or the probe configuration (path, port, command) is incorrect.
- Fix:
- Application Crash: Check pod logs for application-level errors.
Fix the application code.kubectl logs <pod-name> -n <namespace> - Slow Startup: Increase the
initialDelaySecondsin your probe configuration to give the application more time to start before the first probe.livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 30 # Increased from default 0 or 10 periodSeconds: 5 - Incorrect Probe Config: Verify the
path,port, andscheme(for HTTP probes) or thecommand(for exec probes) exactly match what your application expects. If your app listens on port 80, ensure the probe usesport: 80.readinessProbe: exec: command: ["cat", "/tmp/healthy"] # Example for an exec probe initialDelaySeconds: 5 periodSeconds: 2
- Application Crash: Check pod logs for application-level errors.
- Why it works: Liveness probes tell Kubernetes when to restart a container. Readiness probes tell Kubernetes when a pod is ready to receive traffic. If these probes consistently fail, Kubernetes will kill the pod (liveness) or prevent traffic from reaching it (readiness), leading to a "failing" state or it never becoming
Ready.
3. Resource Limits Exceeded
- Diagnosis:
kubectl describe pod <pod-name> -n <namespace>. Look forOOMKilled(Out Of Memory) in the container status, or events related to resource constraints. - Cause: The pod’s container is trying to use more CPU or memory than it’s allowed by its resource requests/limits defined in the pod spec. Minikube has limited resources, so this is common.
- Fix:
- Increase Limits: Adjust the
resources.limitsin your pod spec. For example, to allow more memory:resources: limits: memory: "512Mi" # Increased from, say, 256Mi cpu: "1000m" # Equivalent to 1 CPU core requests: memory: "256Mi" cpu: "500m" - Reduce Usage: Optimize your application to consume fewer resources.
- Increase Minikube Resources: If you consistently hit resource limits, you might need to allocate more resources to the Minikube VM itself.
minikube start --memory 8192 --cpus 4 # Example: 8GB RAM, 4 CPUs
- Increase Limits: Adjust the
- Why it works: Kubernetes enforces resource limits to ensure stable operation. If a container exceeds its memory limit, the kernel terminates it (OOMKilled). If it exceeds its CPU limit, it’s throttled. Adjusting limits or reducing application demand resolves this.
4. Configuration Errors (ConfigMaps/Secrets)
- Diagnosis: Check pod logs (
kubectl logs <pod-name> -n <namespace>) for errors related to missing configuration files, incorrect environment variables, or failed application startup due to bad config. - Cause: The application within the pod relies on a
ConfigMaporSecretthat is missing, incorrectly named, or contains malformed data. - Fix:
- Verify Existence: Ensure the
ConfigMaporSecretexists in the same namespace as the pod.kubectl get configmap <configmap-name> -n <namespace> kubectl get secret <secret-name> -n <namespace> - Check Data: Inspect the contents of the
ConfigMaporSecretfor typos, incorrect keys, or malformed values (e.g., invalid JSON, incorrect base64 encoding for secrets if manually created).kubectl get configmap <configmap-name> -n <namespace> -o yaml kubectl get secret <secret-name> -n <namespace> -o yaml - Correct Pod Spec: Ensure the
volumeMountsorenvsections in your pod spec correctly reference theConfigMaporSecretnames and paths/keys.# Example volume mount from ConfigMap volumes: - name: config-volume configMap: name: my-app-config containers: - name: my-app image: my-image volumeMounts: - name: config-volume mountPath: /etc/config
- Verify Existence: Ensure the
- Why it works: Applications often need external configuration to run correctly. If the configuration provided via
ConfigMapsorSecretsis inaccessible, incorrect, or malformed, the application inside the pod will fail to start or operate.
5. Network Policy Blocking
- Diagnosis: Pods might be
Runningbut not accessible, or they might be stuck in aContainerCreatingstate if they depend on network resources. Checkkubectl describe pod <pod-name> -n <namespace>for events, and then check network policies. - Cause: A
NetworkPolicyin the cluster is preventing the pod from communicating with other necessary pods, services, or the network. This is less common in a default Minikube setup but can occur if you’ve applied network policies. - Fix:
- Inspect Policies: List all network policies in the namespace.
kubectl get networkpolicy -n <namespace> - Review Policy Rules: Examine the rules of any relevant
NetworkPolicyobjects to see if they are too restrictive. For example, a policy might be denying ingress from the pod’s own namespace or denying egress to the Kubernetes API server. - Temporarily Remove Policy: As a test, you can delete a problematic policy.
Caution: This is for debugging only. In production, you should adjust the policy rules to be more permissive as needed.kubectl delete networkpolicy <policy-name> -n <namespace>
- Inspect Policies: List all network policies in the namespace.
- Why it works:
NetworkPoliciesare Kubernetes firewall rules. If a policy incorrectly blocks necessary communication (e.g., a pod can’t reach its DNS server or the API server), it will prevent the pod from functioning or even starting.
6. Pod is Evicted
- Diagnosis:
kubectl get pods -n <namespace> -o wide. Look for pods inEvictedstatus. Then runkubectl describe pod <pod-name> -n <namespace>and look forReason: EvictedandMessage: The node was low on resource: memory/disk/pid. - Cause: The Minikube node (which is a VM) ran out of memory, disk space, or hit a process ID limit. Kubernetes evicted the pod to reclaim resources. This is common on systems with limited host resources or when running many pods in Minikube.
- Fix:
- Free Up Resources on Host: Close other applications on your machine.
- Increase Minikube Resources: Restart Minikube with more memory and/or CPUs:
minikube stop minikube start --memory 8192 --cpus 4 # Example: 8GB RAM, 4 CPUs - Clean Up Unused Resources: Delete old, unused pods, services, or volumes within Minikube.
kubectl delete pod --all -n <namespace> kubectl delete service --all -n <namespace> # For persistent volumes, you might need to delete PVCs and underlying storage.
- Why it works: Eviction is Kubernetes’ way of gracefully handling resource starvation on a node. By increasing the node’s (Minikube VM’s) resources or freeing up space, you prevent the node from becoming starved, thus stopping evictions.
After fixing these, you might encounter CrashLoopBackOff, which means the container is starting, crashing, and then being restarted by Kubernetes over and over.