K3s uses an embedded etcd datastore by default, which is great for single-node setups and quick deployments. However, for production environments, especially those with multiple control-plane nodes, relying on embedded etcd can become a bottleneck and a single point of failure. This guide explains how to configure K3s to use an external PostgreSQL database as its datastore, providing better scalability, resilience, and manageability.

Why PostgreSQL?

PostgreSQL is a powerful, open-source relational database system known for its reliability, robustness, and extensibility. It offers features like ACID compliance, advanced indexing, and replication, making it a suitable candidate for storing K3s cluster state.

Setting up PostgreSQL

First, you need a running PostgreSQL instance. This can be a standalone server, a managed cloud service (like AWS RDS, Google Cloud SQL, Azure Database for PostgreSQL), or a highly available cluster.

  1. Create a Database and User: On your PostgreSQL server, create a dedicated database and a user with privileges to access it.

    -- Connect to your PostgreSQL server as a superuser
    CREATE DATABASE k3s_db;
    CREATE USER k3s_user WITH PASSWORD 'your_strong_password';
    GRANT ALL PRIVILEGES ON DATABASE k3s_db TO k3s_user;
    
  2. Configure PostgreSQL for K3s: Ensure your PostgreSQL instance is accessible from your K3s nodes. You might need to adjust pg_hba.conf and postgresql.conf to allow remote connections from the IP addresses of your K3s servers.

    • pg_hba.conf example:

      # TYPE  DATABASE        USER            ADDRESS                 METHOD
      host    k3s_db          k3s_user        192.168.1.0/24          md5
      

      This allows k3s_user to connect to k3s_db from any IP in the 192.168.1.0/24 subnet using password authentication.

    • postgresql.conf: Ensure listen_addresses is set to * or the specific IP address your K3s nodes will connect to.

      listen_addresses = '*'
      

Configuring K3s to Use PostgreSQL

When installing or starting K3s, you need to provide the connection string to your PostgreSQL database.

  1. Installation (New Cluster): Use the K3S_URL and K3S_TOKEN environment variables along with the --datastore-endpoint flag.

    curl -sfL https://get.k3s.io | sh -s - \
      --datastore-endpoint "postgres://k3s_user:your_strong_password@your_postgres_host:5432/k3s_db?sslmode=disable" \
      --cluster-init
    
    • postgres://k3s_user:your_strong_password@your_postgres_host:5432/k3s_db: This is the standard PostgreSQL connection string format. Replace your_strong_password, your_postgres_host, and 5432 (if using a non-standard port) with your actual PostgreSQL credentials and host.
    • ?sslmode=disable: For simplicity in this example, SSL is disabled. In production, you should configure SSL for secure communication. Use sslmode=require or sslmode=verify-full and ensure your K3s nodes trust the PostgreSQL server’s certificate.
    • --cluster-init: This flag is crucial when initializing a new K3s cluster with an external datastore. It tells K3s to bootstrap the datastore connection and set up the necessary schemas.
  2. Joining a Server Node to an Existing Cluster: If you’re adding a new control-plane node to an existing K3s cluster that already uses PostgreSQL, you’ll use the same --datastore-endpoint and provide the cluster token.

    curl -sfL https://get.k3s.io | K3S_URL="https://your_k3s_server_ip:6443" K3S_TOKEN="your_cluster_token" sh -s - \
      --datastore-endpoint "postgres://k3s_user:your_strong_password@your_postgres_host:5432/k3s_db?sslmode=disable"
    
  3. Starting K3s Manually (Existing Cluster): If K3s is already installed and you want to switch to an external datastore, you’ll modify the K3s service configuration. On your K3s server node, edit /etc/rancher/k3s/config.yaml (or create it if it doesn’t exist) and add the --datastore-endpoint flag.

    server: true
    datastore-endpoint: "postgres://k3s_user:your_strong_password@your_postgres_host:5432/k3s_db?sslmode=disable"
    

    Then, restart the K3s service:

    sudo systemctl restart k3s
    

    Important Note for Existing Clusters: If you are migrating an existing K3s cluster from embedded etcd to PostgreSQL, you must back up your etcd data first. Then, you will need to stop K3s, remove the existing etcd data directory (usually /var/lib/rancher/k3s/server/db/), and then start K3s with the --datastore-endpoint configured. K3s will detect the empty datastore directory and attempt to initialize the schema. This process is destructive to your existing etcd data and should only be performed if you have a backup and understand the implications. For a true migration, you’d typically set up a new cluster with PostgreSQL and then migrate your workloads.

Verification

After configuring K3s to use PostgreSQL, verify the connection.

  1. Check K3s Service Status:

    sudo systemctl status k3s
    

    Look for any errors related to database connection.

  2. Check PostgreSQL Logs: Examine your PostgreSQL server logs for connection attempts from your K3s nodes.

  3. Inspect K3s Resources: Once K3s is running, you should be able to interact with your cluster as usual.

    kubectl get nodes
    kubectl get pods --all-namespaces
    

    If these commands return data, K3s is successfully communicating with your cluster state stored in PostgreSQL.

The Hidden Cost of sslmode=disable

While sslmode=disable makes initial setup simpler, it transmits your PostgreSQL credentials and all cluster data unencrypted over the network. In a production environment, this is a significant security risk. You must configure SSL for your PostgreSQL connection. This involves:

  1. Generating SSL Certificates: Obtain or generate SSL certificates for your PostgreSQL server.

  2. Configuring PostgreSQL for SSL: Update postgresql.conf to enable SSL and specify the certificate and key locations.

  3. Configuring K3s for SSL: Update your K3s datastore endpoint to use sslmode=require or sslmode=verify-full. You may also need to provide the sslrootcert parameter pointing to the CA certificate that signed your PostgreSQL server’s certificate.

    Example with SSL:

    curl -sfL https://get.k3s.io | sh -s - \
      --datastore-endpoint "postgres://k3s_user:your_strong_password@your_postgres_host:5432/k3s_db?sslmode=verify-full&sslrootcert=/path/to/your/ca.crt" \
      --cluster-init
    

    Ensure /path/to/your/ca.crt is accessible on the K3s node.

The next hurdle you’ll likely encounter is managing PostgreSQL’s connection pooling and ensuring it can handle the load from a busy K3s cluster, especially during scaling events or high API server activity.

Want structured learning?

Take the full K3s course →