K3s will automatically deploy any YAML manifest files found in /var/lib/rancher/k3s/server/manifests/.
Here’s how to leverage that behavior to manage your K3s addons:
K3s Auto-Deploy Addons with Manifest Directory
The K3s server component actively watches the /var/lib/rancher/k3s/server/manifests/ directory. When it detects new YAML files, it automatically applies them as Kubernetes manifests, effectively deploying them to your cluster. This provides a simple, declarative way to manage essential addons like network plugins, storage drivers, or custom controllers without needing to manually run kubectl apply.
Let’s see this in action. Imagine we want to deploy a simple Nginx Ingress controller.
First, ensure K3s is running. You can check its status with sudo systemctl status k3s.
Now, create the directory if it doesn’t exist:
sudo mkdir -p /var/lib/rancher/k3s/server/manifests/
Next, create a YAML file for your Nginx Ingress controller. You can find official manifests for many common addons online. For Nginx Ingress, a typical deployment involves a Namespace, Deployment, and Service. Let’s create a file named nginx-ingress.yaml inside the manifests directory.
sudo nano /var/lib/rancher/k3s/server/manifests/nginx-ingress.yaml
Paste the following content into the file:
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app: ingress-nginx
template:
metadata:
labels:
app: ingress-nginx
spec:
containers:
- name: controller
image: registry.k8s.io/ingress-nginx/controller:v1.9.5 # Use a specific, stable version
args:
- /nginx-ingress-controller
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --election-id=ingress-controller-leader
- --controller-class=k8s.io/ingress-nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
type: LoadBalancer # K3s will often handle this via its built-in service load balancer
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
- port: 443
targetPort: https
protocol: TCP
name: https
selector:
app: ingress-nginx
Save and close the file.
The K3s server, which is typically running as a systemd service, will detect this new file. It watches the /var/lib/rancher/k3s/server/manifests/ directory at a regular interval. Upon detection, it effectively runs kubectl apply -f /var/lib/rancher/k3s/server/manifests/nginx-ingress.yaml in the background.
You can verify the deployment by checking the pods and services:
kubectl get pods -n ingress-nginx
kubectl get svc -n ingress-nginx
You should see the ingress-nginx-controller pod running and a service of type LoadBalancer. If you’re running K3s on a single node, K3s’s built-in service load balancer will typically assign an IP address from the node’s network interface.
This auto-deployment mechanism is incredibly useful for ensuring that critical components are always present in your cluster. If you restart K3s, it will re-apply all manifests in this directory, bringing your desired state back online.
The key advantage here is idempotency. If you modify the nginx-ingress.yaml file (e.g., change the number of replicas), K3s will detect the change and apply the update. If you delete the file from the directory, K3s will not automatically delete the deployed resources. This directory is for addition and update, not deletion. To remove an addon deployed this way, you must manually delete the corresponding Kubernetes resources using kubectl delete.
When you need to manage the lifecycle of addons, especially ensuring they are removed cleanly, this directory is best used in conjunction with GitOps tools like Argo CD or Flux CD. These tools can monitor a Git repository containing your manifests, and when changes are pushed, they can update the files in the /var/lib/rancher/k3s/server/manifests/ directory (or directly apply them, depending on configuration), allowing K3s to pick them up.
The K3s server’s manifest directory is a powerful feature, but it’s crucial to understand its limitations. It’s primarily for declarative addition and updates of resources. For more complex lifecycle management, including deletions, consider integrating it with a GitOps workflow.
The next challenge you’ll likely face is managing secrets and configuration maps for these addons, as storing sensitive information directly in the manifests is not recommended.