Kubernetes Secrets Encryption in etcd is a feature that allows you to encrypt sensitive data, like API keys or passwords, stored in etcd.
Here’s how it works in practice. Imagine you have a Secret object in Kubernetes:
apiVersion: v1
kind: Secret
metadata:
name: my-db-credentials
namespace: default
type: Opaque
data:
username: dXNlcm5hbWU= # base64 encoded 'username'
password: cGFzc3dvcmQ= # base64 encoded 'password'
When Kubernetes Secrets Encryption is enabled, this data isn’t stored in etcd as plain text. Instead, it’s encrypted before being written to etcd. When a component (like a Pod) requests this Secret, Kubernetes decrypts it on the fly before returning it.
Let’s see what this looks like under the hood. First, you need to enable the encryption configuration. This is done via a configuration file passed to the kube-apiserver. A common setup uses the aescbc cipher.
Here’s a snippet of what that configuration file (encryption-config.yaml) might look like:
kind: EncryptionConfiguration
apiVersion: apiserver.config.k8s.io/v1
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-32-byte-key>
- identity: {}
The aescbc provider specifies the encryption algorithm (AES in Cipher Block Chaining mode) and the key used for encryption. The keys section contains one or more keys. name is an identifier for the key, and secret is the actual encryption key encoded in base64. This key must be 32 bytes long for AES-256.
To generate a 32-byte key, you can use openssl:
openssl rand -base64 32
This will output a string like v/gq5xY9zM8rJ7xU4pL2fW1sD0vB6kR3nA7fE5yG9zQ=. You’ll replace <base64-encoded-32-byte-key> with this generated string.
The identity: {} provider is crucial. It acts as a fallback and a way to manage key rotation. If you have multiple keys, Kubernetes will try them in order. The identity provider, when listed last, means that any data not encrypted by a previous provider will be stored as is. This is important during key rotation: you can add a new key, re-encrypt secrets, and then remove the old key.
When the kube-apiserver starts, it reads this configuration file. You specify this file using the --encryption-provider-config flag:
kube-apiserver \
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml \
# ... other kube-apiserver flags
With this enabled, if you were to directly inspect etcd (which you generally shouldn’t do, but for demonstration purposes), the data field of your Secret would look like a garbled string, not the base64 encoded username and password.
For example, a secret stored without encryption might show in etcd like:
{"username": "dXNlcm5hbWU=", "password": "cGFzc3dvcmQ="}
But with AES-CBC encryption using key1, it might appear as:
{"username": "e5f3a7b9c1d0e2f4a6b8c0d1e3f5a7b9c1d0e2f4a6b8c0d1e3f5a7b9c1d0e2f4", "password": "f0e1d2c3b4a5f6e7d8c9b0a1f2e3d4c5b6a7f8e9d0c1b2a3f4e5d6c7b8a9f0e1"}
(Note: This is a simplified representation; the actual output is a binary blob, often base64 encoded for storage in etcd’s string fields).
The kube-apiserver intercepts requests for secrets. If it receives a GET request for my-db-credentials, it fetches the encrypted data from etcd, uses key1 to decrypt it, and then returns the decrypted Secret object to the requesting client (e.g., the Kubelet on a node that needs to mount the secret into a Pod).
The most surprising thing about this setup is how seamlessly it operates. Applications and Kubernetes components requesting Secrets don’t need to know that encryption is happening. They get the decrypted data as if it were always plain text, thanks to the API server’s interception and decryption.
The exact levers you control are the choice of encryption provider (e.g., aescbc, kms, secretbox), the specific cipher and mode if applicable, and the keys themselves. Managing these keys securely is paramount. The identity provider is critical for enabling graceful key rotation without downtime.
The next concept you’ll run into is managing multiple encryption keys for rotation or different resource types.