Kubernetes kube-proxy doesn’t just create iptables rules; it rewrites your entire network namespace’s iptables in a way that looks like pure chaos until you realize it’s a highly ordered, albeit complex, state machine.
Let’s see kube-proxy in action. Imagine a simple Service of type ClusterIP with a single Pod backend.
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: my-app
ports:
- protocol: TCP
port: 80
targetPort: 8080
apiVersion: v1
kind: Pod
metadata:
name: my-pod
labels:
app: my-app
spec:
containers:
- name: my-container
image: nginx:latest
ports:
- containerPort: 8080
When you send a packet to my-service:80 (which resolves to a ClusterIP like 10.96.0.10), kube-proxy’s iptables rules intercept it. The first thing you’ll notice is a massive number of iptables rules. Many of them will look like they’re dropping packets, but they’re actually part of the load balancing and service discovery mechanism.
Here’s a glimpse of what you might see on a node after running sudo iptables-save:
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A PREROUTING -m comment --comment "KUBE-SERVICES:..." -j KUBE-SERVICES
-A OUTPUT -m comment --comment "KUBE-SERVICES:..." -j KUBE-SERVICES
COMMIT
*mangle
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A PREROUTING -m comment --comment "KUBE-NODEPORTS:..." -j KUBE-NODEPORTS
-A PREROUTING -m comment --comment "KUBE-LOAD-BALANCER:..." -j KUBE-LOAD-BALANCER
-A POSTROUTING -m comment --comment "KUBE-EXTERNAL-SERVICES:..." -j KUBE-EXTERNAL-SERVICES
COMMIT
*filter
:ACCEPT all -- anywhere anywhere /* KUBE-EXTERNAL-SERVICES */
:REJECT all -- anywhere anywhere /* KUBE-EXTERNAL-SERVICES */
:DROP all -- anywhere anywhere /* KUBE-EXTERNAL-SERVICES */
:ACCEPT all -- anywhere anywhere /* KUBE-NODEPORTS */
:REJECT all -- anywhere anywhere /* KUBE-NODEPORTS */
:DROP all -- anywhere anywhere /* KUBE-NODEPORTS */
:ACCEPT all -- anywhere anywhere /* KUBE-SERVICES */
:REJECT all -- anywhere anywhere /* KUBE-SERVICES */
:DROP all -- anywhere anywhere /* KUBE-SERVICES */
:ACCEPT all -- anywhere anywhere /* KUBE-LOAD-BALANCER */
:REJECT all -- anywhere anywhere /* KUBE-LOAD-BALANCER */
:DROP all -- anywhere anywhere /* KUBE-LOAD-BALANCER */
-A FORWARD -m comment --comment "KUBE-SERVICES-FORWARD:..." -j KUBE-SERVICES-FORWARD
-A FORWARD -m comment --comment "KUBE-NODEPORTS-FORWARD:..." -j KUBE-NODEPORTS-FORWARD
-A FORWARD -m comment --comment "KUBE-LOAD-BALANCER-FORWARD:..." -j KUBE-LOAD-BALANCER-FORWARD
COMMIT
*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# ... and many more chains, specific to your cluster's services and pods
The key chains you’ll see are KUBE-SERVICES, KUBE-NODEPORTS, KUBE-LOAD-BALANCER, and their FORWARD counterparts. kube-proxy uses these chains to implement its functionality.
For our my-service, the PREROUTING chain in the nat table will have a jump to KUBE-SERVICES. KUBE-SERVICES will then contain rules for my-service’s ClusterIP and port. These rules will, in turn, jump to a specific chain for my-service (e.g., KUBE-SVC-ABCDEF1234567890). This service-specific chain will contain rules that use iptables’s random selection to pick one of the backend Pod IPs and ports, and then perform a Destination Network Address Translation (DNAT) to rewrite the destination IP and port to that of the selected Pod.
The FORWARD chain is crucial for traffic originating from other nodes or the internet and destined for a Pod. kube-proxy adds rules here to ensure that these packets are correctly routed and processed by the KUBE-SERVICES and other relevant chains.
The mental model kube-proxy builds is that of a distributed, highly stateful load balancer. It doesn’t just map IPs and ports; it manages the lifecycle of these mappings, reacts to Pod and Service changes, and ensures that traffic is directed to healthy endpoints. The sheer volume of rules is a consequence of kube-proxy needing to handle every possible Service and Pod combination, including different protocols, ports, and load-balancing algorithms.
One of the most surprising things is how kube-proxy manages to be resilient to iptables rule flushes or restarts. When kube-proxy starts or detects a change, it replaces the entire set of rules it manages. It doesn’t incrementally add or delete rules. This is why you’ll often see kube-proxy associated with specific comments in iptables rules (like KUBE-SERVICES) – these are the markers kube-proxy uses to identify and manage its own rule set. If you manually delete a rule managed by kube-proxy, it will likely be recreated on the next sync.
The next concept you’ll run into is how kube-proxy handles NodePort services and external load balancers, which involve additional iptables chains and more complex NAT transformations.