Loki’s "canary" isn’t about detecting failures; it’s about confirming that everything is working, which is a much harder problem.
Let’s see it in action. Imagine you’ve just deployed a new application that’s supposed to be sending logs to Loki. How do you know those logs are actually arriving, and not getting lost somewhere in the pipeline? You can’t just look at a dashboard and assume it’s fine.
Here’s a simplified view of what happens. A log producer (your app) sends logs to an agent (like Promtail). Promtail batches these logs and sends them to Loki’s ingester. The ingester processes them and writes them to object storage. The query engine then reads from storage to serve your requests. At any point, a log can get stuck or dropped.
The canary is a small, deliberate log entry that you inject into your system at the very beginning of this chain. You then try to query for it at the very end. If you can find it, you have high confidence that the entire pipeline is functioning.
Here’s how you’d set it up. First, you need a way to programmatically send a unique log line. This could be a simple echo command run by a cron job, or a small script within your application itself. The key is to include a highly improbable string that you can easily search for.
echo "CANARY_TEST_$(date +%s)_$(openssl rand -hex 8)" | promtail-local-agent --config.file=/etc/promtail/config.yaml
This command uses promtail-local-agent (or your actual Promtail configuration) to send a log line with a timestamp and a random hex string. The timestamp ensures uniqueness, and the random string makes it impossible to accidentally match other logs.
Your Promtail configuration would need to ensure this log is picked up and sent to Loki. A basic scrape_configs entry might look like this:
scrape_configs:
- job_name: canary_sender
static_configs:
- targets:
- localhost
labels:
job: canary_sender
__path__: /path/to/your/canary/log/file # Or however you're feeding it
The critical part is the query. You need to query Loki for that exact, unique string.
logcli query --since 5m 'CANARY_TEST_1678886400_a1b2c3d4e5f67890'
Replace 1678886400_a1b2c3d4e5f67890 with the actual string generated by your canary command. If logcli returns that log line, you’ve confirmed end-to-end delivery.
The "why" is simple: if this unique, high-entropy string made it from your injection point, through the agent, to Loki’s ingester, and then was successfully indexed and stored, then all your other logs of that same type and from that same source are highly likely to have done the same. It’s a smoke test for your entire logging pipeline.
You can automate this by running the injection and query in a script, and failing the script if the query returns no results. This becomes your primary alert for log delivery issues.
The most surprising thing is that many teams don’t do this. They rely on observing their application dashboards and assume logs are flowing. But the log pipeline is a distributed system in itself, with its own failure modes. A canary test is the most robust way to verify its health. It’s not about detecting if a log arrived, but that the mechanism for all logs to arrive is working.
The next step after a successful canary is to ensure the content of your logs is useful, which involves understanding Loki’s indexing and label cardinality.