K3s, designed for simplicity and resource efficiency, can still benefit immensely from robust monitoring, and Prometheus is the de facto standard for this.
Let’s see K3s and Prometheus in action. Imagine a K3s cluster running a simple Nginx deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
Now, let’s deploy Prometheus to scrape metrics from this. We’ll use a standard Prometheus operator setup.
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: prometheus
namespace: monitoring
spec:
chart: prometheus-community/kube-prometheus-stack
repo: https://prometheus-community.github.io/helm-charts
version: "45.2.0" # Use a recent, stable version
values: |
prometheus:
prometheusSpec:
serviceMonitorSelector: {} # Scrape all ServiceMonitors
podMonitorSelector: {} # Scrape all PodMonitors
retention: 10d
resources:
requests:
cpu: 200m
memory: 500Mi
limits:
cpu: 500m
memory: 1Gi
grafana:
enabled: true
adminPassword: "changeme"
alertmanager:
enabled: true
Once deployed, Prometheus will automatically discover and scrape metrics from pods and services that expose them in a Prometheus-compatible format. K3s itself exposes metrics via its API server and controller manager, and applications can be instrumented to expose their own.
The core problem K3s Prometheus solves is providing visibility into a distributed system that’s often deployed in resource-constrained environments. It allows you to understand cluster health, application performance, and resource utilization without the overhead of heavier monitoring solutions.
Internally, Prometheus works by periodically fetching metrics from configured targets (your K3s components and applications) over HTTP. It stores these metrics in a time-series database. Its query language, PromQL, then allows you to slice, dice, and aggregate this data. The ServiceMonitor and PodMonitor Custom Resource Definitions (CRDs) are key here. They tell the Prometheus operator what to scrape.
When you set serviceMonitorSelector: {} and podMonitorSelector: {} in the Helm chart values, you’re telling the Prometheus instance to look for all ServiceMonitor and PodMonitor resources in the cluster, regardless of their namespace. This is a common and convenient way to enable broad monitoring in K3s, as many add-ons and operators (like those for Traefik or your own applications) will automatically create these CRDs for you. The Prometheus operator then configures the Prometheus server’s scrape configuration dynamically based on these CRDs.
The Prometheus server itself is a single binary that can be scaled, but for K3s, the default deployment is usually sufficient. The metrics are exposed on a /metrics endpoint, typically on port 9090 for the Prometheus server itself, and on application-specific ports for your workloads.
Grafana integrates seamlessly for visualization. You’ll typically add Prometheus as a data source in Grafana and then import or create dashboards. Pre-built dashboards for Kubernetes, K3s, and common applications are readily available.
One crucial aspect of Prometheus monitoring in K3s is understanding how ServiceMonitor and PodMonitor CRDs work. These are not Prometheus features themselves but are provided by the Prometheus Operator. When you deploy the kube-prometheus-stack chart, it installs the operator. This operator watches for ServiceMonitor and PodMonitor resources. A ServiceMonitor tells the operator how to find and scrape a Kubernetes Service, while a PodMonitor does the same for individual Pods. For example, a ServiceMonitor might specify that it wants to scrape targets for a service named my-app-service in the default namespace, looking for metrics on port web and path /metrics. The operator then configures the Prometheus instance to add this service as a scrape target.
The next step is usually setting up alerting rules to proactively identify issues before they impact users.