Kubernetes production hardening is less about preventing breaches and more about building an environment that gracefully handles inevitable failures and misconfigurations.

Let’s get our hands dirty with a Kubernetes cluster that’s actually running. Imagine we’re setting up a simple web application.

# Deploy a basic Nginx deployment
kubectl create deployment nginx-app --image=nginx --replicas=3

# Expose it with a ClusterIP service
kubectl expose deployment nginx-app --port=80 --target-port=80 --type=ClusterIP

# Check the service and pods
kubectl get svc,pods -l app=nginx-app

This is the basic scaffolding. We have pods running Nginx, and a service that directs traffic to them internally. But this is far from production-ready. To truly harden this, we need to understand the underlying components and how they interact.

The core of Kubernetes is the control plane: kube-apiserver, etcd, kube-scheduler, and kube-controller-manager. These components talk to each other and manage the state of the cluster. The data plane consists of kubelet and kube-proxy on each node, responsible for running containers and managing network rules.

A Pod is the smallest deployable unit. It can contain one or more containers that share network and storage. A Deployment is a higher-level abstraction that manages Pods and provides declarative updates. A Service provides a stable IP address and DNS name for a set of Pods, abstracting away individual Pod lifecycle.

Here’s where we start hardening. This isn’t a theoretical exercise; these are real commands and configurations you’d use.

1. Secure kube-apiserver Access

Problem: The API server is the front door to your cluster. Unauthenticated or overly permissive access is a direct path to compromise.

Check:

kubectl cluster-info
kubectl get --raw='/version'

This shows basic connectivity. To check auth, we need to look at the API server flags. If running managed Kubernetes (EKS, GKE, AKS), this is handled by the provider. For self-hosted, check kube-apiserver manifest.

Common Causes & Fixes:

  • Anonymous Auth Enabled: kube-apiserver allows unauthenticated requests.

    • Diagnosis: Check kube-apiserver command-line flags for --anonymous-auth=true.
    • Fix: Set --anonymous-auth=false. This is typically done by modifying the static pod manifest for kube-apiserver on control plane nodes (e.g., /etc/kubernetes/manifests/kube-apiserver.yaml).
    • Why it works: Disables the anonymous authentication mechanism, forcing all requests to go through legitimate authentication methods.
  • Excessive RBAC Permissions: Users or service accounts have overly broad roles.

    • Diagnosis: kubectl auth can-i --list (for users) or kubectl auth can-i --list -n <namespace> --as system:serviceaccount:<service-account-name> (for service accounts). Look for roles like cluster-admin.
    • Fix: Create specific Role or ClusterRole objects with least-privilege permissions. For example, a Role that only allows get, list, and watch on pods in a specific namespace.
      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        namespace: default
        name: pod-reader
      rules:
      - apiGroups: [""] # "" indicates the core API group
        resources: ["pods"]
        verbs: ["get", "list", "watch"]
      
      Then bind this role to a user or service account using a RoleBinding or ClusterRoleBinding.
    • Why it works: Enforces the principle of least privilege, limiting what authenticated entities can do.
  • Unencrypted etcd Communication: API server and etcd communicate without TLS.

    • Diagnosis: Check kube-apiserver flags for --etcd-cafile, --etcd-certfile, --etcd-keyfile and etcd flags for --peer-cert-file, --peer-key-file, --cert-file, --key-file.
    • Fix: Ensure all kube-apiserver and etcd instances are configured with valid TLS certificates for communication. This involves generating or obtaining certificates and configuring the respective flags.
    • Why it works: Encrypts traffic between the API server and etcd, preventing eavesdropping and tampering.
  • Unrestricted Admission Controllers: Sensitive admission controllers are not enabled or are misconfigured.

    • Diagnosis: Check kube-apiserver flags for --enable-admission-plugins.
    • Fix: Enable essential admission controllers like PodSecurity, NodeRestriction, NamespaceLifecycle, LimitRanger, and ResourceQuota. For example: --enable-admission-plugins=...,PodSecurity,NodeRestriction,....
    • Why it works: These controllers act as gatekeepers, enforcing security policies and resource constraints before objects are persisted in etcd.
  • API Server Network Exposure: API server is directly accessible from untrusted networks.

    • Diagnosis: Check firewall rules and network security groups for the control plane nodes/load balancer.
    • Fix: Restrict access to the API server’s IP address and port (usually 6443) to only trusted IP ranges (e.g., corporate VPN, specific bastion hosts).
    • Why it works: Limits the attack surface by preventing unauthorized external access.
  • Using kubectl with insecure-skip-tls-verify: Client-side verification is bypassed.

    • Diagnosis: Check ~/.kube/config for insecure-skip-tls-verify: true in the cluster definition.
    • Fix: Remove insecure-skip-tls-verify: true and ensure the certificate-authority-data or certificate-authority points to the correct CA certificate for your cluster.
    • Why it works: Ensures that clients are verifying the identity of the API server, preventing man-in-the-middle attacks.

2. Harden etcd

Problem: etcd is the cluster’s brain, storing all its state. Compromising etcd means compromising the entire cluster.

Check:

# If you have direct access to etcd nodes
etcdctl version

This confirms etcd is running. For security checks, you need to inspect etcd configuration and network access.

Common Causes & Fixes:

  • Unencrypted etcd Communication: As mentioned above, API server and etcd communicate without TLS.

    • Diagnosis: Check etcd flags for --peer-cert-file, --peer-key-file, --cert-file, --key-file.
    • Fix: Ensure etcd is configured with TLS certificates for peer-to-peer and client-server communication.
    • Why it works: Encrypts data in transit between etcd peers and between etcd and the API server.
  • Unrestricted etcd Client Access: Any process can connect to etcd.

    • Diagnosis: Check etcd flags for --listen-client-urls. If it’s http://0.0.0.0:2379 or similar without TLS, it’s too open.
    • Fix: Configure etcd to listen on specific interfaces and enforce client certificate authentication (--client-cert-auth=true). Restrict access via network firewalls to only the kube-apiserver IP.
    • Why it works: Limits etcd access to only authorized components and prevents direct connections from unauthorized sources.
  • No etcd Data Encryption at Rest: etcd data is stored unencrypted on disk.

    • Diagnosis: This is an operational/infrastructure check, not a kubectl command. Check the underlying storage configuration for etcd data directories.
    • Fix: Implement disk-level encryption on the nodes where etcd data resides. For Kubernetes 1.20+, consider using the EncryptionConfiguration feature for etcd data at rest, which can be configured via the API server.
      apiVersion: apiserver.config.k8s.io/v1
      kind: EncryptionConfiguration
      resources:
      - resources: ["secrets"]
        providers:
        - kms:
            name: my-cloud-kms
            cachesize: 10
            endpoint: unix:///var/run/kms-plugin.sock
      
    • Why it works: Protects sensitive data (like secrets) if the underlying storage is physically compromised.
  • Lack of etcd Backups: No regular backups of the etcd state.

    • Diagnosis: Verify if a backup strategy is in place and tested.
    • Fix: Implement a robust backup strategy for etcd. This typically involves periodic snapshots using etcdctl snapshot save and storing them securely off-cluster. Regularly test restore procedures.
    • Why it works: Ensures data recovery in case of catastrophic failure or corruption.
  • Running etcd on Control Plane Nodes with Other Workloads: etcd shares resources and increases the attack surface of control plane nodes.

    • Diagnosis: Check which pods are scheduled on control plane nodes.
    • Fix: Isolate etcd onto dedicated nodes or use managed Kubernetes services that handle this isolation.
    • Why it works: Reduces the blast radius of a compromise on a control plane node and prevents resource contention.

3. Network Policies

Problem: By default, all pods in a Kubernetes cluster can communicate with each other. This is a flat network, which is dangerous.

Check:

kubectl get networkpolicy -A

If this returns no resources, you have no network policies in place.

Common Causes & Fixes:

  • No Network Policy Enforcement: The cluster has no Network Policy controller installed or enabled.

    • Diagnosis: Check your CNI plugin documentation (e.g., Calico, Cilium, Flannel with a separate policy agent). Ensure the CNI supports NetworkPolicy and that it’s enabled. For example, Calico typically requires calico-node and calico-kube-controllers to be running.
    • Fix: Install or enable a CNI plugin that supports NetworkPolicy and has its policy enforcement components running.
    • Why it works: The CNI plugin is responsible for translating NetworkPolicy resources into actual firewall rules on the nodes.
  • Default Allow-All Policy: All pods can talk to each other.

    • Diagnosis: Absence of any NetworkPolicy resources or a policy that explicitly allows all ingress/egress.
    • Fix: Implement a default-deny policy at the namespace level, then create specific NetworkPolicy resources to allow necessary traffic.
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: default-deny-all
        namespace: my-namespace
      spec:
        podSelector: {} # Selects all pods in the namespace
        policyTypes:
        - Ingress
        - Egress
      
      This policy denies all ingress and egress traffic to all pods in my-namespace. Then, create specific policies to allow, for example, ingress from a frontend service to a backend service.
    • Why it works: Establishes a baseline of no communication, forcing explicit allowance of all traffic, thereby reducing the attack surface.
  • Overly Permissive Ingress/Egress Rules: Policies allow too much traffic.

    • Diagnosis: Review your NetworkPolicy YAMLs. Look for podSelector: {} in from or to clauses, or broad namespaceSelector or ipBlock rules.
    • Fix: Refine podSelector to target specific labels, use namespaceSelector to restrict communication to specific namespaces, and use ipBlock only when absolutely necessary for external IPs. For example, allow ingress only from pods with label app: frontend.
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-frontend-to-backend
        namespace: my-namespace
      spec:
        podSelector:
          matchLabels:
            app: backend
        policyTypes:
        - Ingress
        ingress:
        - from:
          - podSelector:
              matchLabels:
                app: frontend
          ports:
          - protocol: TCP
            port: 8080
      
    • Why it works: Granularly defines allowed communication paths, preventing lateral movement of threats.
  • Not Applying Policies to All Namespaces: Some namespaces are left unprotected.

    • Diagnosis: Run kubectl get networkpolicy --all-namespaces and check if all critical namespaces have policies.
    • Fix: Ensure a consistent network policy strategy is applied across all namespaces. Use automation or GitOps to enforce this.
    • Why it works: Creates a uniform security posture across the entire cluster.

4. Pod Security Standards (PSS) / Pod Security Admission (PSA)

Problem: Pods can run with excessive privileges, such as privileged containers or host mounts, which can lead to node compromise.

Check:

kubectl get psp # If using PodSecurityPolicy (deprecated in 1.25)
kubectl describe pss # If using PodSecurityPolicy

For newer versions, this is managed via PodSecurity admission.

Common Causes & Fixes:

  • PodSecurityPolicy Not Configured (Older Kubernetes): No PSPs are defined or enforced.

    • Diagnosis: kubectl get psp. If empty or if restricted PSP is not applied to most users/namespaces.
    • Fix: Define PodSecurityPolicy resources that enforce security best practices (e.g., disallow privileged containers, restrict host mounts, enforce read-only root filesystems). Apply these policies using RoleBindings. Example restricted PSP definition is widely available online.
    • Why it works: Acts as an authorization layer, preventing pods that violate security constraints from being created.
  • PodSecurity Admission Not Enforcing (Kubernetes 1.25+): PSA is enabled but not in enforce mode.

    • Diagnosis: Check kube-apiserver flags for --enable-admission-plugins. Ensure PodSecurity is present. Check the PodSecurity configuration applied to namespaces.
    • Fix: Configure PSA at the namespace level. For example, to enforce the restricted profile:
      # Apply this label to a namespace
      apiVersion: v1
      kind: Namespace
      metadata:
        name: sensitive-namespace
        labels:
          pod-security.kubernetes.io/enforce: restricted # or baseline, privileged
          pod-security.kubernetes.io/enforce-version: v1.28 # specify K8s version
      
      The enforce level can be privileged, baseline, or restricted. restricted is the most secure.
    • Why it works: PSA is a built-in admission controller that enforces predefined security standards for pods based on namespace labels, preventing the creation of insecure pods.
  • privileged: true Containers: Pods are allowed to run as privileged.

    • Diagnosis: kubectl get pods -A -o jsonpath='{.items[*].spec.containers[*].securityContext.privileged}' | grep true.
    • Fix: Ensure your PSPs or PSA labels prevent privileged: true. If a specific pod requires it (rare), document and approve it with extreme caution.
    • Why it works: Prevents containers from gaining full root access to the host operating system.
  • Host Path Mounts: Pods mounting host directories.

    • Diagnosis: kubectl get pods -A -o jsonpath='{.items[*].spec.volumes[*].hostPath}'. Review the output for sensitive paths.
    • Fix: Use PSPs or PSA to disallow hostPath volumes or restrict them to specific, non-sensitive paths.
    • Why it works: Prevents pods from accessing or modifying sensitive host system files.
  • Running as Root User: Containers run as the root user by default.

    • Diagnosis: Check securityContext.runAsUser and securityContext.runAsNonRoot in pod specs.
    • Fix: Configure securityContext in your pod definitions to run containers as a non-root user and set readOnlyRootFilesystem: true.
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        readOnlyRootFilesystem: true
      
    • Why it works: Reduces the impact of a container breakout by limiting the privileges of the running process.

5. Image Security

Problem: Vulnerable or malicious container images are a primary vector for compromise.

Check:

kubectl get pods -A -o jsonpath='{.items[*].spec.containers[*].image}'

This gives you a list of all images used.

Common Causes & Fixes:

  • Using Untrusted Base Images: Images from unknown or unverified sources.

    • Diagnosis: Manually inspect image sources and reputations.
    • Fix: Use trusted, minimal base images (e.g., distroless, alpine, official images). Implement an image registry policy that only allows images from approved repositories.
    • Why it works: Reduces the likelihood of running code with known vulnerabilities or backdoors.
  • Unscanned Images: Images are deployed without vulnerability scanning.

    • Diagnosis: No integration with image scanning tools in your CI/CD pipeline or registry.
    • Fix: Integrate image scanning tools (e.g., Trivy, Clair, Anchore) into your CI/CD pipeline and/or container registry. Fail builds or block deployments if critical vulnerabilities are found.
    • Why it works: Identifies and prevents the deployment of images containing known security flaws.
  • Using :latest Tag: The :latest tag is mutable, making it hard to track and roll back.

    • Diagnosis: Look for image: myapp:latest in your deployment manifests.
    • Fix: Always use specific, immutable image tags (e.g., myapp:v1.2.3, myapp:git-sha-abcdef).
    • Why it works: Ensures consistent deployments and reliable rollbacks, as the image tag always refers to the same immutable artifact.
  • Running Images as Root: Images are built to run processes as root.

    • Diagnosis: Check Dockerfile for USER root or no explicit USER instruction (defaults to root).
    • Fix: In your Dockerfile, use the USER instruction to switch to a non-root user before running your application.
      FROM alpine:latest
      RUN addgroup -S appgroup && adduser -S appuser -G appgroup
      USER appuser
      CMD ["/app/mybinary"]
      
    • Why it works: Aligns with the principle of least privilege for the container process itself.

6. Node Security

Problem: Compromised nodes can grant attackers access to the entire cluster.

Check:

kubectl get nodes -o wide

This shows node IPs and OS images.

Common Causes & Fixes:

  • Unpatched Operating Systems: Nodes running outdated OS versions with known vulnerabilities.

    • Diagnosis: Regularly scan nodes for OS vulnerabilities. Compare node OS versions against security advisories.
    • Fix: Implement a robust patching strategy for your node operating systems. Consider using managed node groups that handle patching automatically.
    • Why it works: Addresses known security flaws in the underlying infrastructure that could be exploited.
  • SSH Access Too Permissive: Wide SSH access to nodes.

    • Diagnosis: Review SSH access control lists (ACLs), firewall rules, and user accounts on nodes.
    • Fix: Restrict SSH access to nodes to only essential personnel and from trusted IP addresses. Use bastion hosts for access. Implement strong SSH key management.
    • Why it works: Limits direct access to the node’s operating system, reducing the attack surface.
  • Kubelet Security: Kubelet on nodes is not properly secured.

    • Diagnosis: Check Kubelet configuration files (e.g., /var/lib/kubelet/config.yaml) and command-line flags. Look for --anonymous-auth=true, --authorization-mode=AlwaysAllow, or --read-only-port being enabled.
    • Fix: Disable anonymous authentication (--anonymous-auth=false), enable authorization (--authorization-mode=Webhook), and ensure the read-only port is disabled (--read-only-port=0). Use TLS for Kubelet communication.
    • Why it works: Prevents unauthorized access to the Kubelet API, which can be used to interact with pods and nodes.
  • No Node Logging and Monitoring: Lack of visibility into node-level events.

    • Diagnosis: No agents or configurations for collecting system logs, kernel logs, or security events from nodes.
    • Fix: Deploy a cluster-wide logging and monitoring solution that collects logs and metrics from all nodes. Integrate security event logging (e.g., auditd, Falco).
    • Why it works: Provides visibility into potential security incidents occurring at the node level, enabling faster detection and response.
  • Running Unnecessary Services on Nodes: Nodes host services beyond Kubernetes requirements.

    • Diagnosis: Use tools like netstat -tulnp or ss -tulnp on nodes to identify listening ports and running services.
    • Fix: Minimize the number of services running on Kubernetes nodes. If possible, run them as pods in the cluster instead.
    • Why it works: Reduces the attack surface of the node by disabling non-essential components.

The next error you’ll likely encounter after implementing these is related to the complexity of managing these configurations at scale, prompting a move towards GitOps or policy-as-code solutions like OPA Gatekeeper.

Want structured learning?

Take the full Kubernetes course →