Loki’s multi-cluster querier lets you query logs from multiple Loki instances as if they were a single, unified log stream.
Let’s see it in action. Imagine two Loki clusters, cluster-a and cluster-b.
# Query logs from cluster-a
loki-query --address http://loki-a.example.com:3100 --query '{job="my-app"}'
# Query logs from cluster-b
loki-query --address http://loki-b.example.com:3100 --query '{job="my-app"}'
Now, with the multi-cluster querier, you can combine these.
First, you need to configure the querier component to know about your other Loki instances. This is typically done via a configuration file.
# querier-config.yaml
auth_enabled: false
multi_cluster:
tenants:
- name: cluster-a
url: http://loki-a.example.com:3100
- name: cluster-b
url: http://loki-b.example.com:3100
querier:
# Default limits for queries
max_concurrent_requests_per_tenant: 100
max_outstanding_requests_per_tenant: 1000
Then, you run the querier binary with this configuration.
loki-querier -config.file=querier-config.yaml
Once running, you can query across clusters. The querier acts as a single endpoint. You’ll use a tool like promtail or logcli configured to point to the querier’s address.
Let’s say the querier is running at http://loki-querier.example.com:3100.
# Query logs from ALL configured clusters
logcli --addr http://loki-querier.example.com:3100 --query '{job="my-app"}'
The logcli command, when pointed at the multi-cluster querier, doesn’t know or care about the individual cluster locations. It sends the query to the querier component. The querier then fans out this query to all configured tenants (your individual Loki clusters). It receives the results from each, merges them, and returns a single, consolidated response to the client.
The primary problem this solves is avoiding the need to manually query each Loki instance and then correlate the results yourself. It presents a unified view, simplifying operations and log analysis, especially in distributed environments. The multi_cluster.tenants section is where you define the name (an arbitrary identifier for your reference) and the url of each individual Loki instance that the querier should include in its queries.
Internally, when a query arrives at the multi-cluster querier, it iterates through the multi_cluster.tenants configuration. For each tenant, it constructs a query to that specific Loki instance’s ingester API. It then uses goroutines to run these queries concurrently. The results from each Loki instance are collected, and the querier performs deduplication and sorting before returning the final aggregated result set. This parallel execution is key to maintaining reasonable query performance across multiple backends.
The querier can also be configured to use a single, shared object store for its index, even if the underlying Loki instances are separate. This is achieved via the index.object_store configuration block. When configured this way, the querier can read the index from a central location, allowing it to determine which Loki instances might contain the requested data before fanning out the query. This can significantly reduce the number of individual Loki instances that need to be queried, improving efficiency for queries that target specific time ranges or labels.
The multi_cluster.tenants configuration doesn’t inherently enforce any label matching or routing logic; it’s a simple list. If you need to route queries based on labels (e.g., logs from region="us-east" go to loki-us-east.example.com and region="eu-west" go to loki-eu-west.example.com), you’d need to implement that logic in your client or a proxy layer before the multi-cluster querier, or potentially build a more sophisticated custom querier.
The next step is usually to explore how to manage query costs and performance more granularly, perhaps by implementing query rate limiting at the querier level or by optimizing the underlying Loki instances for faster retrieval.