The Kubernetes API server is refusing to issue certificates because its trust store doesn’t recognize the Certificate Authority (CA) that signed the API server’s own certificate.
This usually means the kubeadm init or kubeadm join command failed to properly distribute the cluster’s root CA certificate to all components that need to trust it. The kubelet on worker nodes, etcd members, and even the kube-proxy all need to verify the API server’s identity.
Here are the most common reasons this happens and how to fix them:
1. Incorrect kubeadm Configuration
The most frequent culprit is a mismatch or omission in the kubeadm configuration file used during cluster initialization or node joining. Specifically, the certificatesDir might be pointing to the wrong location, or the caCertPath and caKeyPath might not be correctly specified if you’re attempting to use an external CA.
Diagnosis:
Examine the kubeadm-config.yaml file used during kubeadm init or kubeadm join. Look for the ClusterConfiguration section and verify the paths related to certificates. On a node experiencing the error, check the contents of /etc/kubernetes/pki/ca.crt. Does it match the CA certificate expected for your cluster?
Fix:
If kubeadm init failed, rerun it with the correct configuration. If a node failed to join, you’ll likely need to reset and rejoin it.
For kubeadm init:
sudo kubeadm init --config kubeadm-config.yaml
Ensure kubeadm-config.yaml has a correct ClusterConfiguration like this (if using default CA generation):
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.28.0
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
networking:
podSubnet: 10.244.0.0/16
# ... other configurations
If you’re using an external CA, ensure caCertPath and caKeyPath are correctly set and the files exist at those paths.
If a worker node fails to join, reset it and join again:
sudo kubeadm reset -f
sudo kubeadm join <control-plane-ip>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash> --config join-config.yaml
Ensure join-config.yaml (if used) correctly references the cluster’s CA or that the CA certificate was properly distributed.
Why it works: kubeadm is responsible for generating and distributing the cluster’s root CA certificate. If this process is misconfigured, the necessary trust anchor won’t be present where it’s needed.
2. Corrupted or Missing CA Certificate on Nodes
The cluster’s root CA certificate (/etc/kubernetes/pki/ca.crt) might have been accidentally deleted, corrupted, or not properly copied to all nodes that need to communicate with the API server. This is especially common on worker nodes.
Diagnosis:
On the affected node (e.g., a kubelet on a worker node), check if /etc/kubernetes/pki/ca.crt exists and if its content is a valid PEM-encoded certificate. Compare it with the ca.crt on a control plane node. You can use openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text to inspect its details.
Fix: If the CA certificate is missing or corrupted on a worker node, you can copy it from a control plane node. On a control plane node, get the CA:
sudo scp /etc/kubernetes/pki/ca.crt user@worker-node-ip:/tmp/ca.crt
On the worker node, replace the local CA cert:
sudo mv /tmp/ca.crt /etc/kubernetes/pki/ca.crt
sudo chown root:root /etc/kubernetes/pki/ca.crt
sudo chmod 644 /etc/kubernetes/pki/ca.crt
Then, restart the kubelet service:
sudo systemctl restart kubelet
If this is a control plane node experiencing issues, you might need to re-initialize or run kubeadm init phase certs all --config kubeadm-config.yaml to regenerate certificates.
Why it works: The kubelet and other components use /etc/kubernetes/pki/ca.crt to verify the identity of the API server. Ensuring this file is correct and accessible establishes the necessary trust.
3. Outdated kubeadm Version Mismatches
Using different versions of kubeadm for init and join can lead to subtle configuration differences or unexpected behavior in certificate management and distribution.
Diagnosis:
Check the kubeadm version on your control plane node (kubeadm version) and compare it to the kubeadm version on any worker nodes that are failing to join or communicate.
Fix:
Ensure that all nodes intended to be part of the cluster are using the same kubeadm version. If a worker node has a different version, update it to match the control plane:
# Example for Debian/Ubuntu
sudo apt-mark unhold kubeadm
sudo apt-get update
sudo apt-get install -y kubeadm=1.28.0-00 # Replace with your desired version
sudo apt-mark hold kubeadm
After updating kubeadm on the worker, you might need to reset and rejoin the node.
Why it works: kubeadm’s internal logic for certificate generation and configuration can change between versions. A mismatch can cause it to generate or expect configurations that are incompatible between components.
4. Incorrect Permissions on Certificate Files
The certificate files in /etc/kubernetes/pki/ might have incorrect ownership or permissions, preventing the kubelet or other services from reading them.
Diagnosis:
On the affected node, check the permissions and ownership of the files within /etc/kubernetes/pki/. For example:
ls -l /etc/kubernetes/pki/
The ca.crt, apiserver.crt, and apiserver.key should typically be owned by root:root with read permissions for the owner and group (e.g., 644 or 640).
Fix: Correct the permissions and ownership:
sudo chown root:root /etc/kubernetes/pki/ca.crt
sudo chmod 644 /etc/kubernetes/pki/ca.crt
# Repeat for other relevant PKI files if necessary
Then restart the kubelet service:
sudo systemctl restart kubelet
Why it works: Services like kubelet run under specific users (often root or a dedicated kubelet user). If these users cannot read the certificate files, they cannot establish trust with the API server.
5. Firewall Blocking or Network Issues
While less common for a "signed by unknown authority" error specifically, network issues or firewalls could theoretically interfere with the initial handshake or certificate distribution if kubeadm join is relying on network access to fetch necessary trust information.
Diagnosis: Ensure that the worker node can reach the API server’s IP address and port (usually 6443).
# From the worker node
nc -vz <control-plane-ip> 6443
Check iptables or cloud provider security groups to ensure traffic on port 6443 is allowed from worker nodes to the control plane.
Fix:
Adjust firewall rules or security groups to allow traffic on port 6443.
Example using iptables (on control plane, allowing from a worker subnet):
sudo iptables -A INPUT -p tcp --dport 6443 -s <worker-subnet>/24 -j ACCEPT
Then restart kubelet on the worker node.
Why it works: The kubelet needs to establish a TLS connection to the API server. If the network path is blocked, the initial TLS handshake, which includes certificate validation, will fail, potentially leading to misinterpretations of the error.
6. etcd Certificate Issues (Less Common for Kubelet Errors)
If the control plane itself is having issues and the API server is configured to use a self-signed or improperly configured etcd certificate, this could indirectly cause trust issues if components rely on etcd’s CA or if the API server’s certificate was generated with a flawed process rooted in etcd’s setup.
Diagnosis:
Check the etcd logs on the control plane node for any certificate errors. Verify the etcd CA certificate (/etc/kubernetes/pki/etcd/ca.crt) and the API server’s etcd-client.crt/key.
Fix:
This is more complex and often involves re-running kubeadm init phase etcd and kubeadm init phase apiserver-certs or resetting the control plane. It’s usually indicative of a deeper kubeadm init failure.
Why it works: The API server must trust the etcd cluster it’s communicating with. If the etcd certificates are misconfigured, it can cascade into issues with the API server’s own certificate issuance or validation.
After fixing these issues, the next error you’ll likely encounter if there are still underlying problems is a certificados signing error or a failed to get lease error from the kubelet trying to register with the API server.