K3s server tokens are short-lived secrets, and rotating them is a critical security practice to prevent unauthorized cluster access.

This isn’t about just generating a new token; it’s about understanding how K3s manages node joins and the implications of token expiration and compromise.

Here’s how a new node joins a K3s cluster:

  1. Node Bootstrapping: A new node starts up with K3s agent binary.
  2. Token Retrieval: The agent needs to know how to find the server and authenticate. This information is typically provided via the K3S_TOKEN environment variable or a configuration file.
  3. Server Connection: The agent attempts to connect to the K3s server’s API endpoint (defaulting to https://127.0.0.1:6443 if not specified).
  4. Authentication: The agent presents the K3S_TOKEN to the server.
  5. Registration: If the token is valid and not expired, the server registers the new node and issues a TLS certificate for it.

The core of the join process relies on the K3S_TOKEN. This token acts as a shared secret. When a node joins, it uses this token to prove its identity to the server. The server then validates this token and, if successful, proceeds with the node’s registration and certificate issuance. This is why managing its lifecycle is so important.

Rotating the Server Token

The K3s server token is stored in the /var/lib/rancher/k3s/server/node-token file on the server node. To rotate it, you need to generate a new token and then update all existing and future joining nodes with this new token.

1. Generate a New Token:

You can generate a new token by simply creating a new file with a token. A common way is to use openssl to generate a strong random string.

sudo openssl rand -base64 32 | sudo tee /var/lib/rancher/k3s/server/node-token
sudo chmod 644 /var/lib/rancher/k3s/server/node-token

This command generates a 32-byte random string encoded in base64 and writes it to the node-token file, ensuring it has the correct permissions.

2. Update Existing Worker Nodes:

For each existing worker node, you need to update its K3S_TOKEN environment variable or configuration.

  • If using environment variables: SSH into each worker node. Edit the K3s agent service file. The exact location can vary, but it’s often /etc/systemd/system/k3s-agent.service. Find the line that sets K3S_TOKEN (or add it if it’s missing) and update it with the new token. Example:

    # Find the line like this:
    # Environment="K3S_URL=https://your-server-ip:6443"
    # Environment="K3S_TOKEN=YOUR_OLD_TOKEN"
    
    # Change it to:
    Environment="K3S_URL=https://your-server-ip:6443"
    Environment="K3S_TOKEN=YOUR_NEW_TOKEN_HERE"
    

    Then, reload the systemd daemon and restart the k3s-agent:

    sudo systemctl daemon-reload
    sudo systemctl restart k3s-agent
    
  • If using a configuration file (e.g., /etc/rancher/k3s/config.yaml for the agent): SSH into each worker node. Edit the configuration file. Update the token field:

    server: https://your-server-ip:6443
    token: YOUR_NEW_TOKEN_HERE
    

    Then, restart the k3s-agent:

    sudo systemctl restart k3s-agent
    

3. Update New Node Joins:

When you provision new worker nodes, ensure they are configured with the new token from the start. This means updating any scripts, cloud-init configurations, or automated provisioning tools that inject the token.

Why this works: The node-token is the primary secret used by the K3s agent to authenticate with the K3s server during the initial join phase. By rotating this token on the server and then updating all agents to use the new token, you invalidate any potential use of the old token for new joins. The server will reject connections from agents presenting the old token.

Important Considerations:

  • Server URL: Ensure the K3S_URL on your worker nodes correctly points to your K3s server’s API endpoint. If your server’s IP or hostname changes, you’ll need to update this as well.
  • Token Expiration: K3s server tokens do not have a built-in expiration mechanism. Rotation is a manual security process.
  • Security of the Token: The node-token file on the server should have strict permissions (644 or more restrictive) to prevent unauthorized reading.
  • Automated Rotation: For larger or more dynamic environments, consider scripting the token rotation process using K3s’s API or other automation tools.

The Next Problem: After successfully rotating your server token and updating all nodes, you might encounter x509: certificate signed by unknown authority errors on your worker nodes if their certificates were issued using the old token and the server’s CA has been implicitly or explicitly updated.

Want structured learning?

Take the full K3s course →