K3s’s etcd, the distributed key-value store powering your cluster’s state, can be backed up to S3 using a simple, built-in mechanism.
Let’s see it in action. Imagine you have a K3s cluster running and want to back up its etcd data. You’d typically use the k3s etcd-snapshot save command. But to automate this, we’ll make K3s itself trigger these snapshots.
Here’s how you configure K3s to automatically save etcd snapshots to an S3 bucket.
First, you need to configure K3s to know about your S3 credentials and bucket. This is done via environment variables or a configuration file passed to the k3s server process. The most straightforward way for automation is often within a systemd service file or a similar process manager.
Let’s say your S3 bucket is named my-k3s-backups and you’re using AWS credentials. You’d typically set these environment variables before starting the K3s server:
export K3S_ETCD_SNAPSHOTS_S3_BUCKET="my-k3s-backups"
export K3S_ETCD_SNAPSHOTS_S3_ENDPOINT="s3.amazonaws.com" # Or your S3-compatible endpoint
export K3S_ETCD_SNAPSHOTS_S3_ACCESS_KEY_ID="YOUR_ACCESS_KEY_ID"
export K3S_ETCD_SNAPSHOTS_S3_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY"
export K3S_ETCD_SNAPSHOTS_S3_SKIP_SSL="false" # Set to "true" if using a self-signed cert for S3
If you’re using a custom S3 endpoint (like MinIO), you’d specify that in K3S_ETCD_SNAPSHOTS_S3_ENDPOINT. For example, K3S_ETCD_SNAPSHOTS_S3_ENDPOINT="play.min.io:9000". You might also need to set K3S_ETCD_SNAPSHOTS_S3_SKIP_SSL="true" if your MinIO instance uses a self-signed certificate.
With these environment variables set, K3s will periodically take etcd snapshots and upload them. The frequency is controlled by the K3S_ETCD_SNAPSHOTS_SCHEDULE environment variable. The default is 0 * * * *, which means it runs every hour at the top of the hour. You can change this to any valid cron expression, for example, 30 2 * * * to run at 2:30 AM every day.
export K3S_ETCD_SNAPSHOTS_SCHEDULE="30 2 * * *"
K3s also allows you to specify a prefix for your S3 objects, which is useful for organizing backups, especially if you manage multiple clusters.
export K3S_ETCD_SNAPSHOTS_S3_PREFIX="my-cluster/etcd"
This will result in snapshots being stored like s3://my-k3s-backups/my-cluster/etcd/snapshot-2023-10-27T10:30:00Z.br.
The k3s etcd-snapshot save command, when triggered automatically by K3s, uses the etcdctl snapshot save command under the hood. It streams the snapshot data directly to S3 without writing it to disk first. This is efficient and avoids filling up your local disk. The --endpoints flag for etcdctl is automatically populated by K3s based on its internal etcd configuration. The compression used is br (Brotli), which is efficient.
To verify that it’s working, after the scheduled time has passed, you can list the contents of your S3 bucket:
aws s3 ls s3://my-k3s-backups/my-cluster/etcd/ --recursive
You should see files with names like snapshot-YYYY-MM-DDTHH:MM:SSZ.br.
If you need to restore a snapshot, you’d use the k3s etcd-snapshot restore command, pointing it to the S3 location. For example:
k3s etcd-snapshot restore --s3 \
--s3-bucket my-k3s-backups \
--s3-endpoint s3.amazonaws.com \
--s3-access-key-id YOUR_ACCESS_KEY_ID \
--s3-secret-access-key YOUR_SECRET_ACCESS_KEY \
--snapshot-path my-cluster/etcd/snapshot-2023-10-27T10:30:00Z.br \
--name my-restored-cluster
Note that restoring typically requires the K3s server to be stopped, and you’ll need to provide the same S3 credentials.
The mechanism K3s uses to trigger these snapshots is a background goroutine that checks the current time against the K3S_ETCD_SNAPSHOTS_SCHEDULE. When a match occurs, it initiates the save operation. It also handles the S3 upload logic, including chunking and retries for transient network issues.
One detail often overlooked is that the S3 credentials you provide are used by the K3s server process itself. This means if your K3s server is running on a VM or bare metal, the credentials need to be accessible to that process. If you’re running K3s within a Kubernetes deployment (less common for the control plane itself, but possible for stacked etcd), you’d manage these secrets differently. For stacked etcd setups on K3s, the server process directly uses these environment variables.
The system is designed to be robust; if an upload fails due to a temporary network blip, K3s will retry according to its internal logic, up to a certain limit before marking the snapshot as failed.
The next logical step after automating backups is to automate their retrieval and testing, or to configure retention policies on the S3 side to avoid accumulating too many old snapshots.