Taming Eventual Propagation: A Practical Deep Dive

Config lag is the silent killer of Kubernetes deployments, manifesting as a delay between when you update a resource’s configuration and when that change is actually reflected and acted upon by the system.

Let’s watch a ConfigMap change propagate. Imagine we have a simple application pod that reads a configuration file from a ConfigMap.

First, the ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  config.yaml: |
    logLevel: INFO
    featureFlags:
      newUI: false

And the pod that uses it:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app-container
    image: my-app-image:latest
    volumeMounts:
    - name: config-volume
      mountPath: /etc/app/config
  volumes:
  - name: config-volume
    configMap:
      name: app-config

When my-app starts, it mounts /etc/app/config/config.yaml which contains logLevel: INFO and featureFlags.newUI: false.

Now, we update the ConfigMap:

kubectl edit configmap app-config

We change featureFlags.newUI to true.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  config.yaml: |
    logLevel: INFO
    featureFlags:
      newUI: true # Changed from false

Initially, my-app is still running with the old configuration. The change is in the Kubernetes API server’s etcd, but the pod hasn’t "seen" it yet.

Here’s how it all fits together:

1. API Server & etcd: When you kubectl apply or kubectl edit, the change lands in etcd, managed by the API server. This is the source of truth.
2. Controller Manager: Watches etcd for changes to Kubernetes objects (like Deployments, Services, Pods). It reconciles the desired state (what’s in etcd) with the current state of the cluster. For instance, if you scale a Deployment, the Controller Manager notices and creates/deletes Pods.
3. Kubelet: This is the agent running on each node. It watches etcd for Pods that are scheduled to its node. When it sees a Pod definition, it instructs the container runtime (like Docker or containerd) to start the container. Crucially, Kubelet is also responsible for mounting volumes, including ConfigMaps and Secrets.
4. kube-proxy: Manages network rules for Services. It watches etcd for Service and EndpointSlice changes.
5. Application Pod: The application running inside the container. It reads configuration from mounted volumes.

The "lag" happens because the application pod isn’t directly watching etcd. Instead, it’s a chain: etcd -> API Server -> Kubelet -> Filesystem mount -> Application.

When you change app-config, the API server updates etcd. The Kubelet, watching etcd for Pods, eventually notices the my-app Pod’s definition has a reference to an updated ConfigMap. The Kubelet then tells the container runtime to update the mounted volume. This is not instantaneous. The Kubelet has a ConfigMap sync period (defaulting to 1 minute). It polls etcd for changes to ConfigMaps referenced by Pods it manages.

Once the Kubelet updates the mount on the node, the file /etc/app/config/config.yaml on the node’s filesystem is updated. However, the application process inside the container might not automatically re-read this file. Many applications only read configuration at startup. If your application needs to pick up dynamic configuration changes, it must be designed to do so. This often involves watching the file for changes (e.g., using inotify on Linux) or periodically re-reading it.

The surprising thing is that Kubernetes doesn’t magically push file changes into running containers. It updates the mount point on the node, and the application needs to be aware of that. The default behavior for many applications is to treat mounted ConfigMaps and Secrets as read-only, static configuration.

If you want your application to pick up the newUI: true flag without restarting, it needs to actively monitor /etc/app/config/config.yaml for changes and reload its configuration. For example, a Python application might use watchdog to monitor the file, or a Go application might use fsnotify. If the application doesn’t do this, the only way to get the new configuration is to restart the pod, forcing the Kubelet to re-mount the updated ConfigMap.

The next thing you’ll likely encounter is how to manage secrets propagation, which follows a very similar pattern but with added security considerations.