Loki’s "tail" command is a surprisingly powerful way to debug distributed systems by letting you see log output as it’s generated, without needing to wait for indexing or complex queries.
Let’s see it in action. Imagine you have a few microservices running, and you want to watch the logs from just one specific service, say my-app, in its production environment. Here’s how you’d do it:
loki-cli log --since 1h --forward --selector '{app="my-app", env="production"}'
When you run this, your terminal will essentially become a live view of all log lines produced by my-app in the production environment over the last hour, and it will keep streaming new lines as they arrive. You don’t need to know the exact timestamp or content; the selector {app="my-app", env="production"} is all you need.
The magic of Loki’s tailing lies in how Loki itself is architected. Unlike traditional log aggregators that require full indexing of every log line before it’s searchable, Loki indexes only metadata (labels) about the logs. When you issue a query, Loki goes to the object storage where the logs are actually stored and filters based on those labels. The tail command, then, is essentially a persistent query that continuously asks Loki for new data matching your selector, pushing it directly to your client. This is why it’s so fast and efficient for real-time monitoring.
The core problem this solves is the latency between a log event happening and it becoming visible for debugging. In complex, distributed systems with many moving parts, waiting minutes or hours for logs to be indexed and then performing a search can be a productivity killer. Tail allows you to jump into the flow of your application’s execution immediately.
The key levers you control are the selectors and the time range. Selectors are the heart of Loki’s query language, allowing you to filter logs based on the labels attached to them. A common pattern is app="your-service-name", namespace="your-namespace", level="error". The --forward flag is crucial for live tailing; without it, the command would only show logs from the past that match the selector and then exit. The --since flag defines the initial window of historical logs to fetch before starting to stream new ones. You can use relative times like 1h, 15m, or absolute timestamps.
The real power comes from combining selectors. If you have a specific request ID that’s causing trouble across multiple services, you can tail logs for that ID from all services simultaneously. For example, if your request ID is req-12345, and you know your services are labeled with request_id, you could run:
loki-cli log --forward --selector '{request_id="req-12345"}'
This would show you logs from any service that has that request_id label, letting you trace the lifecycle of that specific request across your entire system in real-time.
One aspect that often trips people up is how Loki handles "streams." A stream is defined by a unique combination of labels. When you tail, you’re not just tailing a file; you’re tailing all streams that match your selector. If you have a service that, for some reason, starts emitting logs with a new label combination (e.g., app="my-app", env="production", instance="new-instance-id"), and you were only selecting {app="my-app", env="production"}, Loki will automatically start tailing logs from that new stream as well. This dynamic discovery is part of what makes it so robust.
After you’ve successfully tailed your logs and fixed whatever was breaking, the next thing you’ll likely want to do is filter those tailed logs further by their content.