The Kubernetes API server is refusing connections from the GitLab CI Kubernetes integration, indicating a network or authentication problem between GitLab and your cluster.
Common Causes and Fixes
1. Incorrect Kubernetes API Endpoint:
- Diagnosis: Check the
Kubernetes cluster integrationsettings in GitLab. TheAPI URLmust precisely match the reachable endpoint of your Kubernetes API server. For example, it might look likehttps://192.168.1.100:6443orhttps://kubernetes.example.com:6443. - Fix: Update the
API URLin GitLab’s cluster integration settings to the correct, publicly or internally resolvable address of your Kubernetes API server. - Why it works: GitLab needs to know the exact network address to initiate a connection.
2. Firewall Blocking Access:
- Diagnosis: If your GitLab instance and Kubernetes cluster are in different network segments, a firewall might be blocking traffic on the Kubernetes API port (default 6443). Use
telnetorncfrom the GitLab server (or a machine that can reach it) to test connectivity to the Kubernetes API endpoint and port.telnet <kubernetes_api_endpoint> 6443 # or nc -zv <kubernetes_api_endpoint> 6443 - Fix: Configure your firewall to allow inbound traffic from your GitLab server’s IP address(es) to the Kubernetes API server’s IP address(es) on port 6443.
- Why it works: Ensures the network path is open for communication.
3. Invalid or Expired TLS Certificates:
- Diagnosis: The TLS certificate presented by the Kubernetes API server might be untrusted by GitLab, expired, or have a hostname mismatch. Check the GitLab logs for TLS handshake errors. You can also test the certificate validity from the GitLab server using
openssl s_client.
Look for "Verify return code: 0 (ok)" and ensure the certificate’s subject alternative names (SANs) include the hostname you are using in theopenssl s_client -connect <kubernetes_api_endpoint>:6443 -servername <kubernetes_api_endpoint>API URL. - Fix: Ensure your Kubernetes API server’s certificate is valid, trusted by GitLab’s CA bundle, and that its SANs include the
API URLhostname. If using self-signed certificates, you may need to add the CA certificate to GitLab’s trusted certificate store or configure GitLab to trust it specifically. - Why it works: GitLab must trust the identity of the Kubernetes API server it’s connecting to.
4. Incorrect RBAC Permissions:
- Diagnosis: The
Service Accountused by GitLab within Kubernetes might lack the necessary permissions to interact with the API server. While this usually results in specific API call failures rather than a complete "not reachable" error, it’s worth checking if other symptoms are present. Thekubectlcommand executed from the GitLab server (if you have direct access and credentials) to the cluster should succeed.kubectl --server=<kubernetes_api_endpoint> --certificate-authority=<path_to_ca.crt> --token=<your_token> get pods - Fix: Ensure the
Service Accountassociated with the GitLab integration has aClusterRoleorRolewith sufficient permissions (e.g.,get,list,watchonpods,deployments,services, etc.). This is typically managed by thegitlab-managed-appsnamespace or similar. - Why it works: The Service Account acts as the identity for GitLab within Kubernetes, and it needs explicit authorization to perform actions.
5. Network Policy Restricting Access:
- Diagnosis: Kubernetes
NetworkPolicyobjects might be in place that prevent pods in certain namespaces (including potentially the one where GitLab’s agents run, or the API server itself) from communicating with each other. Check forNetworkPolicyresources in thekube-systemnamespace or any namespace where your Kubernetes API server is exposed.kubectl get networkpolicies --all-namespaces - Fix: Adjust or create
NetworkPolicyresources to allow traffic from your GitLab integration’s components to the Kubernetes API server on port 6443. - Why it works: Network Policies are Kubernetes-native firewalls that control pod-to-pod communication.
6. DNS Resolution Issues:
- Diagnosis: If you are using a hostname for your Kubernetes API endpoint (e.g.,
kubernetes.example.com), GitLab might not be able to resolve it to an IP address. Test DNS resolution from the GitLab server.ping kubernetes.example.com # or dig kubernetes.example.com - Fix: Ensure your GitLab server can resolve the hostname of your Kubernetes API endpoint via its configured DNS servers. This might involve updating
/etc/resolv.confon the GitLab server or configuring your network’s DNS. - Why it works: Hostnames need to be translated into IP addresses for network connections.
7. GitLab Runner Network Configuration:
- Diagnosis: If you are using GitLab Runners that are themselves part of the Kubernetes cluster, their network configuration or the cluster’s egress rules might prevent them from reaching the API server if it’s exposed externally or on a different network segment than expected.
- Fix: Ensure the
gitlab-runnerpods can reach the Kubernetes API endpoint. This might involve adjusting CNI configurations, egress firewall rules, or Service definitions. - Why it works: The Runner needs to communicate with the API server to schedule jobs.
After resolving these, you might encounter "Unauthorized" errors if RBAC permissions are still not fully aligned.