The most surprising truth about Loki’s deployment modes is that the "distributed" mode, while offering higher availability and scalability, often introduces more complexity than it solves for many teams, and the single binary can be surprisingly robust.
Let’s see Loki in action. Imagine a Kubernetes cluster with a few microservices generating logs. We want to collect these logs, store them, and query them efficiently.
Here’s a snippet of a Loki configuration for a single binary deployment, typically running as a single pod in Kubernetes:
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
path_prefix: /loki
storage:
filesystem:
directory: /loki/data
shared_store:
boltdb:
directory: /loki/boltdb
schema_config:
configs:
- from: 2020-10-24
store: boltdb
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
In this setup, a single Loki process handles ingestion, indexing, and querying. For a small to medium-sized team or application set, this can be incredibly simple to manage. When a log line comes in, Loki receives it, indexes it (creating references to its location), and writes it to the configured storage. When you query, Loki consults its index to find the relevant log chunks and retrieves them.
Now, let’s contrast that with a distributed setup. This typically involves multiple instances of Loki components: ingesters, distributors, queriers, and an index store. A common architecture might look like this:
- Distributor: Receives logs from agents (like Promtail), validates them, and routes them to the appropriate ingester.
- Ingester: Receives log streams, chunks them, and writes them to object storage (like S3 or GCS) and the index store.
- Querier: Receives queries, fetches data from the index store and object storage, and aggregates results.
- Index Store: A dedicated database (like Cassandra, DynamoDB, or even a relational DB with appropriate setup) for storing index data.
Here’s a conceptual configuration for a distributed querier, highlighting its interaction with other components:
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
path_prefix: /loki
storage:
s3:
bucketnames: loki-logs
common_ingester_address: http://loki-ingester:3100
ha_peers:
- loki-querier-0:3100
- loki-querier-1:3100
schema_config:
configs:
- from: 2020-10-24
store: cassandra
object_store: s3
schema: v11
index:
prefix: index_
period: 24h
The "distributed" mode is designed to handle massive scale and high availability. If one ingester fails, others can pick up the load. If one querier is overloaded, you can add more to distribute the query traffic. The distributor acts as a smart load balancer, ensuring logs are spread out.
The core problem Loki solves is providing a cost-effective, scalable, and queryable log aggregation system without the need for complex indexing of every single log line, unlike traditional full-text search engines. It achieves this by indexing only metadata (labels) and then chunking log lines. This makes both storage and querying significantly cheaper and faster when you know what labels to filter by.
When you choose between single binary and distributed, think about your team’s operational capacity and your scale. For many teams, the single binary mode, perhaps run with a replica set in Kubernetes for basic fault tolerance, is sufficient. The "distributed" mode implies managing separate components, a dedicated index store (which can be a significant operational burden itself), and inter-component communication.
The most common pitfall when setting up distributed Loki is underestimating the operational overhead of the index store. Whether you use Cassandra, DynamoDB, or another backend, it requires its own tuning, scaling, and maintenance. Many teams find themselves spending more time managing the index store than Loki itself, especially when using self-hosted options. A single binary with an object store backend and a simple index (like BoltDB or even DynamoDB if you want managed scaling) can often provide a much lower barrier to entry and surprisingly good performance for many use cases.
The next challenge you’ll likely face is optimizing query performance and cost as your data volume grows, regardless of your deployment mode.