Promtail’s main job is to find logs in Kubernetes and send them to Loki.
Here’s a pod that’s not sending logs to Loki, and how to fix it.
apiVersion: v1
kind: Pod
metadata:
name: my-app
labels:
app: my-app
spec:
containers:
- name: my-app-container
image: ubuntu:latest
command: ["/bin/sh", "-c", "while true; do echo 'Hello from my-app $(date)'; sleep 5; done"]
volumeMounts:
- name: logs
mountPath: /var/log
initContainers:
- name: log-creator
image: ubuntu:latest
command: ["/bin/sh", "-c", "while true; do echo 'Init log message $(date)' >> /var/log/init.log; sleep 10; done"]
volumeMounts:
- name: logs
mountPath: /var/log
volumes:
- name: logs
emptyDir: {}
This pod uses an emptyDir volume to capture logs from both the main container and an init container. Promtail, by default, doesn’t know how to find these logs.
The problem is Promtail needs to be told where to look for the logs. It doesn’t magically discover files written to arbitrary paths within a pod. It relies on configuration that maps Kubernetes metadata to log files.
Here’s how to tell Promtail to scrape logs from this pod. We’ll use a ServiceMonitor for Prometheus Operator to manage the Promtail configuration.
First, ensure you have Prometheus Operator installed and that it’s managing Promtail. You should have a Promtail custom resource that looks something like this:
apiVersion: monitoring.coreos.com/v1
kind: Promtail
metadata:
name: promtail
namespace: monitoring
spec:
selector:
matchLabels:
app.kubernetes.io/name: promtail
template:
metadata:
labels:
app.kubernetes.io/name: promtail
spec:
containers:
- name: promtail
image: grafana/promtail:2.4.1 # Use your Promtail version
args:
- "-config.file=/etc/promtail/config/config.yaml"
- "-promtail.grpc-listen-port=9090"
- "-promtail.http-listen-port=9090"
volumeMounts:
- name: config
mountPath: /etc/promtail/config
volumes:
- name: config
secret:
secretName: promtail-config-promtail # This secret will hold our config.yaml
The key is the config.yaml that Promtail will load. This is where we define how to find and process logs. We’ll create a Secret for this.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: promtail-config-promtail
namespace: monitoring
type: Opaque
data:
config.yaml: |
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki.loki.svc.cluster.local:3100/loki/api/v1/push
scrape_configs:
- job_name: kubernetes-pod-logs
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
- source_labels: [__meta_kubernetes_pod_container_name]
target_label: container
- action: keep
regex: my-app # Only scrape logs from pods with label app=my-app
source_labels:
- __meta_kubernetes_pod_label_app
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: __tmp_namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: __tmp_pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: __tmp_container
- target_label: filename
replacement: /var/log/containers/${1}.log # This is a placeholder, we'll fix it next
regex: __meta_kubernetes_pod_container_name
- action: scrape
target_label: __path__
replacement: /var/log/containers/${1}.log # This is a placeholder, we'll fix it next
regex: __meta_kubernetes_pod_container_name
# --- This is the CRITICAL part for multi-container/init containers ---
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: __tmp_container_name
- action: replace
source_labels:
- __meta_kubernetes_pod_container_image
target_label: image
- action: replace
source_labels:
- __meta_kubernetes_pod_annotation_grafana_loki_io_container_log_path
target_label: __path__
regex: (.+) # Capture the log path from the annotation
# --- End of CRITICAL part ---
# This is how we tell Promtail to look into the emptyDir volume
# We need to match the container name to the log file name.
# For init containers, Promtail usually sees them as "init-container-name"
# and for regular containers, "container-name".
# We'll use annotations to explicitly define the log path.
# For the main container, it's often /var/log/<container-name>.log
# For init containers, it might be /var/log/<init-container-name>.log
# Let's refine this with explicit annotations on the pod.
# For the init container, we want to scrape /var/log/init.log
# For the main container, we want to scrape /var/log/my-app-container.log (or similar)
# The default behavior tries to guess, but explicit is better.
# We'll use annotations on the pod to guide Promtail.
# The annotation `grafana.loki.io/container-log-path` is key here.
# It tells Promtail exactly which file to tail for a given container.
# We'll apply this annotation to our pod.
EOF
Now, let’s annotate the pod to tell Promtail where the logs are.
apiVersion: v1
kind: Pod
metadata:
name: my-app
labels:
app: my-app
annotations:
# For the init container
grafana.loki.io/init-container-log-path: "/var/log/init.log"
# For the main container
grafana.loki.io/my-app-container-log-path: "/var/log/my-app-container.log"
spec:
containers:
- name: my-app-container
image: ubuntu:latest
command: ["/bin/sh", "-c", "while true; do echo 'Hello from my-app $(date)'; sleep 5; done"]
volumeMounts:
- name: logs
mountPath: /var/log
initContainers:
- name: log-creator
image: ubuntu:latest
command: ["/bin/sh", "-c", "while true; do echo 'Init log message $(date)' >> /var/log/init.log; sleep 10; done"]
volumeMounts:
- name: logs
mountPath: /var/log
volumes:
- name: logs
emptyDir: {}
Let’s adjust the Promtail config.yaml to properly use these annotations. The __meta_kubernetes_pod_annotation_grafana_loki_io_container_log_path is what we need to leverage. However, this annotation is for a specific container. Promtail’s kubernetes_sd_configs with role: pod and role: container are the standard ways.
For emptyDir volumes shared across containers, the most robust way is to use the __path__ relabeling target and explicitly point to the log file within the pod’s filesystem.
Here’s the corrected config.yaml within the secret. We’ll use ServiceMonitor to inject this config into Promtail.
First, create the ServiceMonitor resource. This tells Prometheus Operator to configure Promtail.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: promtail-app-logs
namespace: monitoring # Namespace where Promtail is deployed
spec:
selector:
matchLabels:
app: my-app # Label on the pod we want to scrape
namespaceSelector:
matchNames:
- default # Namespace where your pod is running
endpoints:
- port: http # This port is not actually used for logs, but required by ServiceMonitor
targetPort: 9080 # The HTTP port Promtail listens on for metrics, not logs
relabelings:
- source_labels: [__meta_kubernetes_pod_container_name]
target_label: container
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
- source_labels: [__meta_kubernetes_pod_label_app]
target_label: app
# This is where the magic happens for log paths
# We'll use the annotation to define the log path for each container.
# Promtail's __path__ target is key here.
# We need to tell Promtail to look for files based on the annotation.
# The annotation `grafana.loki.io/container-log-path` is the standard.
# We need to map container names to these annotations.
# For init containers, the source_labels might be different.
# Let's simplify by using a common annotation for all containers.
# The `grafana.loki.io/container-log-path` annotation is the most flexible.
# It expects a JSON map like: `{"container-name": "/path/to/log.log"}`
# For our pod, we'll use this:
# grafana.loki.io/container-log-path: '{"log-creator": "/var/log/init.log", "my-app-container": "/var/log/my-app-container.log"}'
# We'll apply this annotation to the pod.
- action: replace
source_labels:
- __meta_kubernetes_pod_annotation_grafana_loki_io_container_log_path
target_label: __path__
regex: '.*"my-app-container":\s*(".*?")(?:,.*)?' # Extract path for my-app-container
replacement: $1
- action: replace
source_labels:
- __meta_kubernetes_pod_annotation_grafana_loki_io_container_log_path
target_label: __path__
regex: '.*"log-creator":\s*(".*?")(?:,.*)?' # Extract path for log-creator
replacement: $1
- action: replace
source_labels:
- __meta_kubernetes_pod_annotation_grafana_loki_io_container_log_path
target_label: __path__
regex: '.*"log-creator":\s*(".*?")' # Extract path for log-creator (if it's the only one)
replacement: $1
Now, update your pod with the correct annotation:
apiVersion: v1
kind: Pod
metadata:
name: my-app
labels:
app: my-app
annotations:
# This annotation maps container names to their log file paths within the pod.
grafana.loki.io/container-log-path: '{"log-creator": "/var/log/init.log", "my-app-container": "/var/log/my-app-container.log"}'
spec:
containers:
- name: my-app-container
image: ubuntu:latest
command: ["/bin/sh", "-c", "while true; do echo 'Hello from my-app $(date)'; sleep 5; done"]
volumeMounts:
- name: logs
mountPath: /var/log
initContainers:
- name: log-creator
image: ubuntu:latest
command: ["/bin/sh", "-c", "while true; do echo 'Init log message $(date)' >> /var/log/init.log; sleep 10; done"]
volumeMounts:
- name: logs
mountPath: /var/log
volumes:
- name: logs
emptyDir: {}
With this setup, Promtail, managed by Prometheus Operator via the ServiceMonitor, will discover your my-app pod. It will then inspect the grafana.loki.io/container-log-path annotation. For each container it finds within that pod, it will attempt to match the container name to an entry in the JSON map. If a match is found, it will use the specified path as the log file to tail. The regex in the ServiceMonitor’s relabelings section is crucial for extracting the correct log path for each container.
The next error you’ll hit is Promtail complaining about missing log files if the annotation is malformed or the path is incorrect.