Kubernetes’s default approach to secrets management is to store them unencrypted in etcd, which is often a non-starter for security-conscious organizations.
Let’s see how Flux, a GitOps tool, can dynamically pull secrets from HashiCorp Vault and inject them into Kubernetes pods without ever storing them directly in Git.
First, ensure you have Vault running and accessible, and that Flux is installed in your Kubernetes cluster. Flux needs a way to authenticate with Vault. The most common and secure method is using Kubernetes Service Accounts.
Here’s a typical Vault setup for this scenario:
# policies/kubernetes.hcl
path "secret/data/my-app/*" {
capabilities = ["read"]
}
This policy grants read access to secrets under the secret/data/my-app/ path.
Next, you’ll create a Kubernetes Secret that Flux will use to authenticate with Vault. This Secret contains the JWT token issued by the Kubernetes auth method in Vault.
# vault-auth-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: vault-auth
namespace: flux-system # Or wherever your Flux controller runs
type: Opaque
stringData:
token: <your-vault-client-token-here> # This token is obtained by Vault's Kubernetes auth method
Self-correction: While a direct client token can work, the more robust GitOps pattern is to use Vault’s Kubernetes auth method. Flux, running as a ServiceAccount in Kubernetes, will authenticate with Vault using its ServiceAccount token. Vault then validates this token and issues a short-lived token for Flux to use.
To enable Vault’s Kubernetes auth method:
vault auth enable kubernetes
vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host="https://<kubernetes-api-server-host>:<port>" \
kubernetes_ca_cert="$(cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt)"
Now, you configure Flux’s Kustomization resource to use the Vault integration. This involves a SecretProvider configuration.
# flux-kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: my-app-secrets
namespace: flux-system
spec:
interval: 5m
path: ./clusters/my-cluster/apps/my-app
sourceRef:
kind: GitRepository
name: flux-system
# ... other spec fields
secretGenerator:
- name: my-app-secrets # This is the Kubernetes Secret Flux will create
behavior: merge
files:
# Reference to the actual secret in Vault
db_password.txt: 'vault:secret/data/my-app/database?key=password'
api_key.txt: 'vault:secret/data/my-app/api?key=key'
In this Kustomization, secretGenerator tells Flux to create a Kubernetes Secret named my-app-secrets. The files section specifies that db_password.txt and api_key.txt should be populated from Vault. The syntax vault:secret/data/my-app/database?key=password points to a specific secret in Vault.
Flux, when processing this Kustomization, will:
- Authenticate with Vault using its
ServiceAccount(assuming the Kubernetes auth method is configured and Flux has permission). - Read the specified secrets from Vault.
- Create or update a Kubernetes
Secretnamedmy-app-secretsin the same namespace as theKustomization(or as specified). - The contents of the Vault secret keys will be the values of the files within the Kubernetes
Secret.
Your application deployment (Deployment, Pod, etc.) can then reference this generated Secret just like any other Kubernetes Secret.
# my-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
namespace: default
spec:
template:
spec:
containers:
- name: app
image: my-app-image
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: my-app-secrets # The Secret generated by Flux
key: db_password.txt
- name: API_KEY
valueFrom:
secretKeyRef:
name: my-app-secrets
key: api_key.txt
The my-app-secrets Kubernetes Secret is created by Flux and contains the actual secret values, but it’s never committed to Git. Flux pulls it from Vault on each reconciliation.
The most surprising thing about this setup is that Flux doesn’t store the secrets in its own internal state or in etcd directly. It acts as a transient bridge, fetching them from Vault at reconciliation time and injecting them into the Kubernetes Secret object that your applications then consume. This means the secret values are only ever present in memory within the Flux controller and then in the Kubernetes Secret object itself, which is encrypted at rest in etcd if you’ve configured that.
When you update a secret in Vault, Flux will detect the change during its next reconciliation interval (e.g., 5m) and automatically update the my-app-secrets Kubernetes Secret. Your application pods will then pick up the new secret values if they are configured to watch for changes (e.g., by restarting the pod or using tools like reloader).
The next hurdle you’ll likely encounter is managing the lifecycle of these secrets, specifically ensuring that applications can gracefully reload their configurations when secrets change without requiring a full pod restart.