The kubelet on your Minikube node is failing to connect to CoreDNS, preventing pods from resolving internal Kubernetes service names.
Cause 1: CoreDNS Pod Not Running or Crashing
Diagnosis: Check the status of the CoreDNS pods.
kubectl get pods -n kube-system -l k8s-app=kube-dns
Look for pods in CrashLoopBackOff, Error, or Pending states. If no pods are listed, the deployment might be broken.
Fix: If pods are crashing, inspect their logs for clues.
kubectl logs <coredns-pod-name> -n kube-system
Common reasons for crashing include misconfigurations in the Corefile or resource constraints. Often, simply deleting and recreating the pods will fix transient issues:
kubectl delete pod <coredns-pod-name> -n kube-system
Kubernetes will automatically restart them. If the problem persists, check the Corefile for syntax errors.
Why it works: CoreDNS is deployed as a set of pods. If these pods are unhealthy, the DNS service within the cluster breaks. Recreating the pods forces a fresh start, and if the underlying issue is temporary, it resolves.
Cause 2: CoreDNS Service Not Available
Diagnosis:
Verify that the kube-dns service exists and is correctly configured.
kubectl get service -n kube-system -l k8s-app=kube-dns
You should see a service named kube-dns with ClusterIP and ports including 53/UDP and 53/TCP.
Fix: If the service is missing or misconfigured, it needs to be recreated. This is less common as it’s managed by the Kubernetes control plane. However, if you’ve manually altered resources, you might need to:
# This is a simplified example, actual service definition is more complex
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
labels:
k8s-app: kube-dns
spec:
selector:
k8s-app: kube-dns
ports:
- name: dns
port: 53
protocol: UDP
targetPort: 53
- name: dns-tcp
port: 53
protocol: TCP
targetPort: 53
EOF
Ensure the selector correctly matches the labels on the CoreDNS pods.
Why it works: Pods rely on a Kubernetes Service to provide a stable IP address and port for DNS resolution. If this service is absent or points to the wrong pods, DNS queries won’t reach CoreDNS.
Cause 3: Incorrect kubelet DNS Configuration
Diagnosis:
Check the kubelet configuration on the Minikube node. The kubelet is responsible for telling pods where to find DNS.
# SSH into the Minikube node
minikube ssh
# Then check kubelet config (location varies by installation)
# Common location: /var/lib/kubelet/config.yaml or /etc/kubernetes/kubelet.conf
cat /var/lib/kubelet/config.yaml | grep clusterDNS
You should see clusterDNS pointing to the ClusterIP of the kube-dns service (e.g., 10.96.0.10).
Fix:
If clusterDNS is incorrect or missing, you need to update the kubelet configuration. This often involves restarting the kubelet service.
# On the Minikube VM (after minikube ssh)
sudo systemctl edit kubelet --full
# In the editor, find/add the following under 'clusterDNS':
# clusterDNS:
# - 10.96.0.10 # This IP must match your kube-dns service ClusterIP
# Save and exit, then restart kubelet
sudo systemctl restart kubelet
You might need to restart pods after this for them to pick up the new DNS settings.
Why it works: Pods inherit their DNS server configuration from the kubelet. If the kubelet is configured with the wrong DNS server IP, pods will attempt to query an incorrect address.
Cause 4: Network Policy Blocking DNS Traffic
Diagnosis: Check if any NetworkPolicies are in place that might be blocking egress traffic from pods to the CoreDNS service IP on port 53.
kubectl get networkpolicies --all-namespaces
If you find policies, examine their rules.
Fix:
If a NetworkPolicy is blocking DNS, you’ll need to adjust it to allow egress to the kube-dns service.
# Example: Allow egress to the kube-dns service IP on port 53
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: default # Or the namespace of your application pods
spec:
podSelector: {} # Applies to all pods in the namespace
policyTypes:
- Egress
egress:
- to:
- ipBlock:
# Replace with your actual kube-dns ClusterIP
cidr: 10.96.0.0/12 # A broad range often used for service IPs
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
EOF
Ensure the cidr and ports match your cluster’s DNS setup.
Why it works: NetworkPolicies enforce network segmentation. A misconfigured policy can inadvertently block essential cluster communications, including DNS lookups.
Cause 5: Firewall on Host or VM Interfering
Diagnosis:
On the host machine running Minikube, check if any local firewall rules (e.g., ufw, firewalld, iptables) are blocking traffic to the Minikube VM’s IP address or the Docker bridge network. Also, check within the Minikube VM for similar issues.
# On the host:
sudo ufw status
sudo firewall-cmd --list-all
sudo iptables -L
# On the Minikube VM (after minikube ssh):
sudo ufw status
sudo firewall-cmd --list-all
sudo iptables -L
Fix: If a firewall is blocking traffic, add rules to allow communication on UDP/TCP port 53 between your host, the Minikube VM, and potentially the Docker network interface used by Minikube.
# Example for ufw on host (adjust IP/interface as needed):
sudo ufw allow in on docker0 to any port 53
sudo ufw allow out on docker0 from any port 53
# Example for firewalld on host:
sudo firewall-cmd --permanent --add-port=53/udp
sudo firewall-cmd --permanent --add-port=53/tcp
sudo firewall-cmd --reload
The exact commands depend heavily on your host OS and firewall setup.
Why it works: Firewalls act as gatekeepers for network traffic. If they block DNS-related packets, pods will be unable to reach the CoreDNS server, even if CoreDNS itself is functioning correctly.
Cause 6: Corrupted resolv.conf in Pods
Diagnosis:
Exec into a pod that is having resolution issues and inspect its resolv.conf.
kubectl exec -it <pod-name> -- cat /etc/resolv.conf
The nameserver entry should point to the ClusterIP of the kube-dns service. If it’s incorrect or missing, the pod won’t know where to send DNS queries.
Fix:
This is usually a symptom of an underlying kubelet or CNI (Container Network Interface) issue. Restarting the kubelet (as in Cause 3) or recreating the pod is often the quickest way to get a fresh resolv.conf. If it persists, you might be looking at a CNI plugin problem.
kubectl delete pod <pod-name>
Why it works: Each pod has its own resolv.conf file that dictates its DNS resolution configuration. If this file is corrupted or points to the wrong nameserver, DNS resolution will fail for that specific pod.
You’ll likely hit ImagePullBackOff if your cluster cannot reach external registries due to DNS issues.