Loki’s auto-discovery for Kubernetes pod logs isn’t just about finding logs; it’s about Loki actively becoming part of your Kubernetes control plane to identify and ingest logs from any pod that matches your defined criteria.
Let’s see it in action. Imagine you have a deployment of Nginx pods.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx # This label is key for discovery
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
Now, in Loki, you configure a ServiceMonitor (or a PodMonitor if you’re using the Prometheus Operator’s PodMonitor CRD) to tell Loki how to find these logs.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: loki-pod-logs
labels:
release: loki # Assuming this is the release name of your Loki Helm chart
spec:
selector:
matchLabels:
app: nginx # Loki will look for pods with this label
namespaceSelector:
matchNames:
- default # Or the specific namespace where your pods run
endpoints:
- port: http-metrics # This is not for logs, but for Prometheus metrics
interval: 30s
path: /metrics # Again, for Prometheus metrics
# The magic for logs happens below, implicitly through the pod labels
The crucial part here is that Loki, via its integration with the Prometheus Operator (or a similar mechanism if you’re not using the operator), watches Kubernetes API for pods matching the selector in your ServiceMonitor or PodMonitor. When it sees a pod with app: nginx in the default namespace, it automatically configures a log collection target for that pod.
The promtail agent, which is typically deployed as a DaemonSet in your Kubernetes cluster, is what actually collects the logs. promtail is configured to watch for these discovered targets. It uses the pod’s labels and annotations to determine which logs to scrape and how to label them in Loki.
For example, promtail might look at a pod’s labels like app: nginx, kubernetes_namespace: default, and kubernetes_pod_name: nginx-deployment-abcde, and automatically attach these as labels to the log streams it sends to Loki. This means when you query Loki, you can filter by app="nginx" or kubernetes_namespace="default" to see logs from all your Nginx pods.
The mental model to build is that Loki doesn’t poll for logs; it subscribes to a stream of log targets defined by Kubernetes labels. The Prometheus Operator (or a similar discovery mechanism) acts as the intermediary, translating your ServiceMonitor or PodMonitor definitions into concrete log collection tasks for promtail.
Internally, promtail discovers pods that match the criteria defined in the ServiceMonitor/PodMonitor. It then uses the Kubernetes API to get the container runtime information (like the log file path for a given container in a pod) and tail those files. The labels on the pod and its containers are then automatically applied as labels to the log stream in Loki, enabling powerful filtering and querying.
The exact levers you control are primarily through the selector fields in your ServiceMonitor or PodMonitor CRDs. These selectors dictate which pods Loki’s discovery mechanism should pay attention to. You can match on specific labels, namespaces, or even use more complex set-based selectors. Annotations on pods can also be used to fine-tune promtail’s behavior, such as specifying custom log file paths or modifying labels.
What most people miss is that the ServiceMonitor or PodMonitor itself doesn’t directly configure log scraping. Its primary role is discovery for Prometheus metrics. Loki leverages this same discovery mechanism and extends it to logs. The actual log scraping configuration is implicitly handled by promtail based on the discovered pod and its containers, using the labels and annotations as its guide.
Once you have auto-discovery working, the next step is to ensure your logs are structured correctly for efficient querying.