Kubernetes production hardening is less about preventing breaches and more about building an environment that gracefully handles inevitable failures and misconfigurations.
Let’s get our hands dirty with a Kubernetes cluster that’s actually running. Imagine we’re setting up a simple web application.
# Deploy a basic Nginx deployment
kubectl create deployment nginx-app --image=nginx --replicas=3
# Expose it with a ClusterIP service
kubectl expose deployment nginx-app --port=80 --target-port=80 --type=ClusterIP
# Check the service and pods
kubectl get svc,pods -l app=nginx-app
This is the basic scaffolding. We have pods running Nginx, and a service that directs traffic to them internally. But this is far from production-ready. To truly harden this, we need to understand the underlying components and how they interact.
The core of Kubernetes is the control plane: kube-apiserver, etcd, kube-scheduler, and kube-controller-manager. These components talk to each other and manage the state of the cluster. The data plane consists of kubelet and kube-proxy on each node, responsible for running containers and managing network rules.
A Pod is the smallest deployable unit. It can contain one or more containers that share network and storage. A Deployment is a higher-level abstraction that manages Pods and provides declarative updates. A Service provides a stable IP address and DNS name for a set of Pods, abstracting away individual Pod lifecycle.
Here’s where we start hardening. This isn’t a theoretical exercise; these are real commands and configurations you’d use.
1. Secure kube-apiserver Access
Problem: The API server is the front door to your cluster. Unauthenticated or overly permissive access is a direct path to compromise.
Check:
kubectl cluster-info
kubectl get --raw='/version'
This shows basic connectivity. To check auth, we need to look at the API server flags. If running managed Kubernetes (EKS, GKE, AKS), this is handled by the provider. For self-hosted, check kube-apiserver manifest.
Common Causes & Fixes:
-
Anonymous Auth Enabled:
kube-apiserverallows unauthenticated requests.- Diagnosis: Check
kube-apiservercommand-line flags for--anonymous-auth=true. - Fix: Set
--anonymous-auth=false. This is typically done by modifying the static pod manifest forkube-apiserveron control plane nodes (e.g.,/etc/kubernetes/manifests/kube-apiserver.yaml). - Why it works: Disables the anonymous authentication mechanism, forcing all requests to go through legitimate authentication methods.
- Diagnosis: Check
-
Excessive RBAC Permissions: Users or service accounts have overly broad roles.
- Diagnosis:
kubectl auth can-i --list(for users) orkubectl auth can-i --list -n <namespace> --as system:serviceaccount:<service-account-name>(for service accounts). Look for roles likecluster-admin. - Fix: Create specific
RoleorClusterRoleobjects with least-privilege permissions. For example, aRolethat only allowsget,list, andwatchonpodsin a specific namespace.
Then bind this role to a user or service account using aapiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: default name: pod-reader rules: - apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["get", "list", "watch"]RoleBindingorClusterRoleBinding. - Why it works: Enforces the principle of least privilege, limiting what authenticated entities can do.
- Diagnosis:
-
Unencrypted etcd Communication: API server and etcd communicate without TLS.
- Diagnosis: Check
kube-apiserverflags for--etcd-cafile,--etcd-certfile,--etcd-keyfileandetcdflags for--peer-cert-file,--peer-key-file,--cert-file,--key-file. - Fix: Ensure all
kube-apiserverandetcdinstances are configured with valid TLS certificates for communication. This involves generating or obtaining certificates and configuring the respective flags. - Why it works: Encrypts traffic between the API server and etcd, preventing eavesdropping and tampering.
- Diagnosis: Check
-
Unrestricted Admission Controllers: Sensitive admission controllers are not enabled or are misconfigured.
- Diagnosis: Check
kube-apiserverflags for--enable-admission-plugins. - Fix: Enable essential admission controllers like
PodSecurity,NodeRestriction,NamespaceLifecycle,LimitRanger, andResourceQuota. For example:--enable-admission-plugins=...,PodSecurity,NodeRestriction,.... - Why it works: These controllers act as gatekeepers, enforcing security policies and resource constraints before objects are persisted in etcd.
- Diagnosis: Check
-
API Server Network Exposure: API server is directly accessible from untrusted networks.
- Diagnosis: Check firewall rules and network security groups for the control plane nodes/load balancer.
- Fix: Restrict access to the API server’s IP address and port (usually 6443) to only trusted IP ranges (e.g., corporate VPN, specific bastion hosts).
- Why it works: Limits the attack surface by preventing unauthorized external access.
-
Using
kubectlwithinsecure-skip-tls-verify: Client-side verification is bypassed.- Diagnosis: Check
~/.kube/configforinsecure-skip-tls-verify: truein the cluster definition. - Fix: Remove
insecure-skip-tls-verify: trueand ensure thecertificate-authority-dataorcertificate-authoritypoints to the correct CA certificate for your cluster. - Why it works: Ensures that clients are verifying the identity of the API server, preventing man-in-the-middle attacks.
- Diagnosis: Check
2. Harden etcd
Problem: etcd is the cluster’s brain, storing all its state. Compromising etcd means compromising the entire cluster.
Check:
# If you have direct access to etcd nodes
etcdctl version
This confirms etcd is running. For security checks, you need to inspect etcd configuration and network access.
Common Causes & Fixes:
-
Unencrypted
etcdCommunication: As mentioned above, API server andetcdcommunicate without TLS.- Diagnosis: Check
etcdflags for--peer-cert-file,--peer-key-file,--cert-file,--key-file. - Fix: Ensure
etcdis configured with TLS certificates for peer-to-peer and client-server communication. - Why it works: Encrypts data in transit between
etcdpeers and betweenetcdand the API server.
- Diagnosis: Check
-
Unrestricted
etcdClient Access: Any process can connect toetcd.- Diagnosis: Check
etcdflags for--listen-client-urls. If it’shttp://0.0.0.0:2379or similar without TLS, it’s too open. - Fix: Configure
etcdto listen on specific interfaces and enforce client certificate authentication (--client-cert-auth=true). Restrict access via network firewalls to only thekube-apiserverIP. - Why it works: Limits
etcdaccess to only authorized components and prevents direct connections from unauthorized sources.
- Diagnosis: Check
-
No
etcdData Encryption at Rest:etcddata is stored unencrypted on disk.- Diagnosis: This is an operational/infrastructure check, not a
kubectlcommand. Check the underlying storage configuration foretcddata directories. - Fix: Implement disk-level encryption on the nodes where
etcddata resides. For Kubernetes 1.20+, consider using theEncryptionConfigurationfeature foretcddata at rest, which can be configured via the API server.apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: ["secrets"] providers: - kms: name: my-cloud-kms cachesize: 10 endpoint: unix:///var/run/kms-plugin.sock - Why it works: Protects sensitive data (like secrets) if the underlying storage is physically compromised.
- Diagnosis: This is an operational/infrastructure check, not a
-
Lack of
etcdBackups: No regular backups of theetcdstate.- Diagnosis: Verify if a backup strategy is in place and tested.
- Fix: Implement a robust backup strategy for
etcd. This typically involves periodic snapshots usingetcdctl snapshot saveand storing them securely off-cluster. Regularly test restore procedures. - Why it works: Ensures data recovery in case of catastrophic failure or corruption.
-
Running
etcdon Control Plane Nodes with Other Workloads:etcdshares resources and increases the attack surface of control plane nodes.- Diagnosis: Check which pods are scheduled on control plane nodes.
- Fix: Isolate
etcdonto dedicated nodes or use managed Kubernetes services that handle this isolation. - Why it works: Reduces the blast radius of a compromise on a control plane node and prevents resource contention.
3. Network Policies
Problem: By default, all pods in a Kubernetes cluster can communicate with each other. This is a flat network, which is dangerous.
Check:
kubectl get networkpolicy -A
If this returns no resources, you have no network policies in place.
Common Causes & Fixes:
-
No Network Policy Enforcement: The cluster has no Network Policy controller installed or enabled.
- Diagnosis: Check your CNI plugin documentation (e.g., Calico, Cilium, Flannel with a separate policy agent). Ensure the CNI supports NetworkPolicy and that it’s enabled. For example, Calico typically requires
calico-nodeandcalico-kube-controllersto be running. - Fix: Install or enable a CNI plugin that supports
NetworkPolicyand has its policy enforcement components running. - Why it works: The CNI plugin is responsible for translating
NetworkPolicyresources into actual firewall rules on the nodes.
- Diagnosis: Check your CNI plugin documentation (e.g., Calico, Cilium, Flannel with a separate policy agent). Ensure the CNI supports NetworkPolicy and that it’s enabled. For example, Calico typically requires
-
Default Allow-All Policy: All pods can talk to each other.
- Diagnosis: Absence of any
NetworkPolicyresources or a policy that explicitly allows all ingress/egress. - Fix: Implement a default-deny policy at the namespace level, then create specific
NetworkPolicyresources to allow necessary traffic.
This policy denies all ingress and egress traffic to all pods inapiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: my-namespace spec: podSelector: {} # Selects all pods in the namespace policyTypes: - Ingress - Egressmy-namespace. Then, create specific policies to allow, for example, ingress from a frontend service to a backend service. - Why it works: Establishes a baseline of no communication, forcing explicit allowance of all traffic, thereby reducing the attack surface.
- Diagnosis: Absence of any
-
Overly Permissive Ingress/Egress Rules: Policies allow too much traffic.
- Diagnosis: Review your
NetworkPolicyYAMLs. Look forpodSelector: {}infromortoclauses, or broadnamespaceSelectororipBlockrules. - Fix: Refine
podSelectorto target specific labels, usenamespaceSelectorto restrict communication to specific namespaces, and useipBlockonly when absolutely necessary for external IPs. For example, allow ingress only from pods with labelapp: frontend.apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-frontend-to-backend namespace: my-namespace spec: podSelector: matchLabels: app: backend policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: frontend ports: - protocol: TCP port: 8080 - Why it works: Granularly defines allowed communication paths, preventing lateral movement of threats.
- Diagnosis: Review your
-
Not Applying Policies to All Namespaces: Some namespaces are left unprotected.
- Diagnosis: Run
kubectl get networkpolicy --all-namespacesand check if all critical namespaces have policies. - Fix: Ensure a consistent network policy strategy is applied across all namespaces. Use automation or GitOps to enforce this.
- Why it works: Creates a uniform security posture across the entire cluster.
- Diagnosis: Run
4. Pod Security Standards (PSS) / Pod Security Admission (PSA)
Problem: Pods can run with excessive privileges, such as privileged containers or host mounts, which can lead to node compromise.
Check:
kubectl get psp # If using PodSecurityPolicy (deprecated in 1.25)
kubectl describe pss # If using PodSecurityPolicy
For newer versions, this is managed via PodSecurity admission.
Common Causes & Fixes:
-
PodSecurityPolicyNot Configured (Older Kubernetes): No PSPs are defined or enforced.- Diagnosis:
kubectl get psp. If empty or ifrestrictedPSP is not applied to most users/namespaces. - Fix: Define
PodSecurityPolicyresources that enforce security best practices (e.g., disallow privileged containers, restrict host mounts, enforce read-only root filesystems). Apply these policies usingRoleBindings. ExamplerestrictedPSP definition is widely available online. - Why it works: Acts as an authorization layer, preventing pods that violate security constraints from being created.
- Diagnosis:
-
PodSecurityAdmission Not Enforcing (Kubernetes 1.25+): PSA is enabled but not inenforcemode.- Diagnosis: Check
kube-apiserverflags for--enable-admission-plugins. EnsurePodSecurityis present. Check thePodSecurityconfiguration applied to namespaces. - Fix: Configure PSA at the namespace level. For example, to enforce the
restrictedprofile:
The# Apply this label to a namespace apiVersion: v1 kind: Namespace metadata: name: sensitive-namespace labels: pod-security.kubernetes.io/enforce: restricted # or baseline, privileged pod-security.kubernetes.io/enforce-version: v1.28 # specify K8s versionenforcelevel can beprivileged,baseline, orrestricted.restrictedis the most secure. - Why it works: PSA is a built-in admission controller that enforces predefined security standards for pods based on namespace labels, preventing the creation of insecure pods.
- Diagnosis: Check
-
privileged: trueContainers: Pods are allowed to run as privileged.- Diagnosis:
kubectl get pods -A -o jsonpath='{.items[*].spec.containers[*].securityContext.privileged}' | grep true. - Fix: Ensure your PSPs or PSA labels prevent
privileged: true. If a specific pod requires it (rare), document and approve it with extreme caution. - Why it works: Prevents containers from gaining full root access to the host operating system.
- Diagnosis:
-
Host Path Mounts: Pods mounting host directories.
- Diagnosis:
kubectl get pods -A -o jsonpath='{.items[*].spec.volumes[*].hostPath}'. Review the output for sensitive paths. - Fix: Use PSPs or PSA to disallow
hostPathvolumes or restrict them to specific, non-sensitive paths. - Why it works: Prevents pods from accessing or modifying sensitive host system files.
- Diagnosis:
-
Running as Root User: Containers run as the root user by default.
- Diagnosis: Check
securityContext.runAsUserandsecurityContext.runAsNonRootin pod specs. - Fix: Configure
securityContextin your pod definitions to run containers as a non-root user and setreadOnlyRootFilesystem: true.securityContext: runAsUser: 1000 runAsGroup: 1000 runAsNonRoot: true readOnlyRootFilesystem: true - Why it works: Reduces the impact of a container breakout by limiting the privileges of the running process.
- Diagnosis: Check
5. Image Security
Problem: Vulnerable or malicious container images are a primary vector for compromise.
Check:
kubectl get pods -A -o jsonpath='{.items[*].spec.containers[*].image}'
This gives you a list of all images used.
Common Causes & Fixes:
-
Using Untrusted Base Images: Images from unknown or unverified sources.
- Diagnosis: Manually inspect image sources and reputations.
- Fix: Use trusted, minimal base images (e.g.,
distroless,alpine, official images). Implement an image registry policy that only allows images from approved repositories. - Why it works: Reduces the likelihood of running code with known vulnerabilities or backdoors.
-
Unscanned Images: Images are deployed without vulnerability scanning.
- Diagnosis: No integration with image scanning tools in your CI/CD pipeline or registry.
- Fix: Integrate image scanning tools (e.g., Trivy, Clair, Anchore) into your CI/CD pipeline and/or container registry. Fail builds or block deployments if critical vulnerabilities are found.
- Why it works: Identifies and prevents the deployment of images containing known security flaws.
-
Using
:latestTag: The:latesttag is mutable, making it hard to track and roll back.- Diagnosis: Look for
image: myapp:latestin your deployment manifests. - Fix: Always use specific, immutable image tags (e.g.,
myapp:v1.2.3,myapp:git-sha-abcdef). - Why it works: Ensures consistent deployments and reliable rollbacks, as the image tag always refers to the same immutable artifact.
- Diagnosis: Look for
-
Running Images as Root: Images are built to run processes as root.
- Diagnosis: Check
DockerfileforUSER rootor no explicitUSERinstruction (defaults to root). - Fix: In your
Dockerfile, use theUSERinstruction to switch to a non-root user before running your application.FROM alpine:latest RUN addgroup -S appgroup && adduser -S appuser -G appgroup USER appuser CMD ["/app/mybinary"] - Why it works: Aligns with the principle of least privilege for the container process itself.
- Diagnosis: Check
6. Node Security
Problem: Compromised nodes can grant attackers access to the entire cluster.
Check:
kubectl get nodes -o wide
This shows node IPs and OS images.
Common Causes & Fixes:
-
Unpatched Operating Systems: Nodes running outdated OS versions with known vulnerabilities.
- Diagnosis: Regularly scan nodes for OS vulnerabilities. Compare node OS versions against security advisories.
- Fix: Implement a robust patching strategy for your node operating systems. Consider using managed node groups that handle patching automatically.
- Why it works: Addresses known security flaws in the underlying infrastructure that could be exploited.
-
SSH Access Too Permissive: Wide SSH access to nodes.
- Diagnosis: Review SSH access control lists (ACLs), firewall rules, and user accounts on nodes.
- Fix: Restrict SSH access to nodes to only essential personnel and from trusted IP addresses. Use bastion hosts for access. Implement strong SSH key management.
- Why it works: Limits direct access to the node’s operating system, reducing the attack surface.
-
Kubelet Security: Kubelet on nodes is not properly secured.
- Diagnosis: Check Kubelet configuration files (e.g.,
/var/lib/kubelet/config.yaml) and command-line flags. Look for--anonymous-auth=true,--authorization-mode=AlwaysAllow, or--read-only-portbeing enabled. - Fix: Disable anonymous authentication (
--anonymous-auth=false), enable authorization (--authorization-mode=Webhook), and ensure the read-only port is disabled (--read-only-port=0). Use TLS for Kubelet communication. - Why it works: Prevents unauthorized access to the Kubelet API, which can be used to interact with pods and nodes.
- Diagnosis: Check Kubelet configuration files (e.g.,
-
No Node Logging and Monitoring: Lack of visibility into node-level events.
- Diagnosis: No agents or configurations for collecting system logs, kernel logs, or security events from nodes.
- Fix: Deploy a cluster-wide logging and monitoring solution that collects logs and metrics from all nodes. Integrate security event logging (e.g., auditd, Falco).
- Why it works: Provides visibility into potential security incidents occurring at the node level, enabling faster detection and response.
-
Running Unnecessary Services on Nodes: Nodes host services beyond Kubernetes requirements.
- Diagnosis: Use tools like
netstat -tulnporss -tulnpon nodes to identify listening ports and running services. - Fix: Minimize the number of services running on Kubernetes nodes. If possible, run them as pods in the cluster instead.
- Why it works: Reduces the attack surface of the node by disabling non-essential components.
- Diagnosis: Use tools like
The next error you’ll likely encounter after implementing these is related to the complexity of managing these configurations at scale, prompting a move towards GitOps or policy-as-code solutions like OPA Gatekeeper.