K3s uses an embedded etcd datastore by default, which is great for single-node setups and quick deployments. However, for production environments, especially those with multiple control-plane nodes, relying on embedded etcd can become a bottleneck and a single point of failure. This guide explains how to configure K3s to use an external PostgreSQL database as its datastore, providing better scalability, resilience, and manageability.
Why PostgreSQL?
PostgreSQL is a powerful, open-source relational database system known for its reliability, robustness, and extensibility. It offers features like ACID compliance, advanced indexing, and replication, making it a suitable candidate for storing K3s cluster state.
Setting up PostgreSQL
First, you need a running PostgreSQL instance. This can be a standalone server, a managed cloud service (like AWS RDS, Google Cloud SQL, Azure Database for PostgreSQL), or a highly available cluster.
-
Create a Database and User: On your PostgreSQL server, create a dedicated database and a user with privileges to access it.
-- Connect to your PostgreSQL server as a superuser CREATE DATABASE k3s_db; CREATE USER k3s_user WITH PASSWORD 'your_strong_password'; GRANT ALL PRIVILEGES ON DATABASE k3s_db TO k3s_user; -
Configure PostgreSQL for K3s: Ensure your PostgreSQL instance is accessible from your K3s nodes. You might need to adjust
pg_hba.confandpostgresql.confto allow remote connections from the IP addresses of your K3s servers.-
pg_hba.confexample:# TYPE DATABASE USER ADDRESS METHOD host k3s_db k3s_user 192.168.1.0/24 md5This allows
k3s_userto connect tok3s_dbfrom any IP in the192.168.1.0/24subnet using password authentication. -
postgresql.conf: Ensurelisten_addressesis set to*or the specific IP address your K3s nodes will connect to.listen_addresses = '*'
-
Configuring K3s to Use PostgreSQL
When installing or starting K3s, you need to provide the connection string to your PostgreSQL database.
-
Installation (New Cluster): Use the
K3S_URLandK3S_TOKENenvironment variables along with the--datastore-endpointflag.curl -sfL https://get.k3s.io | sh -s - \ --datastore-endpoint "postgres://k3s_user:your_strong_password@your_postgres_host:5432/k3s_db?sslmode=disable" \ --cluster-initpostgres://k3s_user:your_strong_password@your_postgres_host:5432/k3s_db: This is the standard PostgreSQL connection string format. Replaceyour_strong_password,your_postgres_host, and5432(if using a non-standard port) with your actual PostgreSQL credentials and host.?sslmode=disable: For simplicity in this example, SSL is disabled. In production, you should configure SSL for secure communication. Usesslmode=requireorsslmode=verify-fulland ensure your K3s nodes trust the PostgreSQL server’s certificate.--cluster-init: This flag is crucial when initializing a new K3s cluster with an external datastore. It tells K3s to bootstrap the datastore connection and set up the necessary schemas.
-
Joining a Server Node to an Existing Cluster: If you’re adding a new control-plane node to an existing K3s cluster that already uses PostgreSQL, you’ll use the same
--datastore-endpointand provide the cluster token.curl -sfL https://get.k3s.io | K3S_URL="https://your_k3s_server_ip:6443" K3S_TOKEN="your_cluster_token" sh -s - \ --datastore-endpoint "postgres://k3s_user:your_strong_password@your_postgres_host:5432/k3s_db?sslmode=disable" -
Starting K3s Manually (Existing Cluster): If K3s is already installed and you want to switch to an external datastore, you’ll modify the K3s service configuration. On your K3s server node, edit
/etc/rancher/k3s/config.yaml(or create it if it doesn’t exist) and add the--datastore-endpointflag.server: true datastore-endpoint: "postgres://k3s_user:your_strong_password@your_postgres_host:5432/k3s_db?sslmode=disable"Then, restart the K3s service:
sudo systemctl restart k3sImportant Note for Existing Clusters: If you are migrating an existing K3s cluster from embedded etcd to PostgreSQL, you must back up your etcd data first. Then, you will need to stop K3s, remove the existing etcd data directory (usually
/var/lib/rancher/k3s/server/db/), and then start K3s with the--datastore-endpointconfigured. K3s will detect the empty datastore directory and attempt to initialize the schema. This process is destructive to your existing etcd data and should only be performed if you have a backup and understand the implications. For a true migration, you’d typically set up a new cluster with PostgreSQL and then migrate your workloads.
Verification
After configuring K3s to use PostgreSQL, verify the connection.
-
Check K3s Service Status:
sudo systemctl status k3sLook for any errors related to database connection.
-
Check PostgreSQL Logs: Examine your PostgreSQL server logs for connection attempts from your K3s nodes.
-
Inspect K3s Resources: Once K3s is running, you should be able to interact with your cluster as usual.
kubectl get nodes kubectl get pods --all-namespacesIf these commands return data, K3s is successfully communicating with your cluster state stored in PostgreSQL.
The Hidden Cost of sslmode=disable
While sslmode=disable makes initial setup simpler, it transmits your PostgreSQL credentials and all cluster data unencrypted over the network. In a production environment, this is a significant security risk. You must configure SSL for your PostgreSQL connection. This involves:
-
Generating SSL Certificates: Obtain or generate SSL certificates for your PostgreSQL server.
-
Configuring PostgreSQL for SSL: Update
postgresql.confto enable SSL and specify the certificate and key locations. -
Configuring K3s for SSL: Update your K3s datastore endpoint to use
sslmode=requireorsslmode=verify-full. You may also need to provide thesslrootcertparameter pointing to the CA certificate that signed your PostgreSQL server’s certificate.Example with SSL:
curl -sfL https://get.k3s.io | sh -s - \ --datastore-endpoint "postgres://k3s_user:your_strong_password@your_postgres_host:5432/k3s_db?sslmode=verify-full&sslrootcert=/path/to/your/ca.crt" \ --cluster-initEnsure
/path/to/your/ca.crtis accessible on the K3s node.
The next hurdle you’ll likely encounter is managing PostgreSQL’s connection pooling and ensuring it can handle the load from a busy K3s cluster, especially during scaling events or high API server activity.