The most surprising thing about Loki’s distributed mode is that you’re not just scaling Loki, you’re scaling its independent components to meet your specific needs.
Let’s see this in action. Imagine you’ve got a spiky workload: during a deployment, your logs surge for 10 minutes, then drop back to normal. You don’t want to over-provision your entire Loki cluster for those 10 minutes. Instead, you can scale only the components that are struggling.
Here’s a typical Loki distributed setup:
- Ingesters: These receive logs from your agents (like Promtail), process them, and write them to object storage. If your log volume is high, you’ll scale these.
- Distributor: This acts as a frontend for the ingesters, receiving all incoming logs, validating them, and sending them to the appropriate ingester. It’s the first line of defense.
- Queriers: These handle log queries from users or tools, fetching data from object storage and the index. If your query load is heavy, you’ll scale these.
- Index (Cassandra/DynamoDB): Loki uses a key-value store for its indexes. This is crucial for fast lookups. If your indexing performance is a bottleneck, you scale this database.
- Object Storage (S3/GCS/MinIO): This is where your actual log data is stored. This component is usually managed by the cloud provider or object storage solution, but its performance impacts read/write speeds.
Let’s say you’re seeing slow writes to Loki, and your loki-ingester pods are hitting 90% CPU.
# Example deployment for ingesters
apiVersion: apps/v1
kind: Deployment
metadata:
name: loki-ingester
namespace: loki
spec:
replicas: 3 # We'll scale this up
selector:
matchLabels:
app: loki-ingester
template:
metadata:
labels:
app: loki-ingester
spec:
containers:
- name: ingester
image: grafana/loki:2.9.0
ports:
- containerPort: 3100
# ... other configuration
To fix slow writes, you’d increase the replica count:
# Scale up ingesters
kubectl scale deployment loki-ingester --replicas=6 -n loki
This increases the number of ingester instances, distributing the write load across more machines. Each ingester now handles fewer logs, reducing CPU pressure and improving write throughput.
Conversely, if your users complain about slow query responses, and your loki-querier pods are showing high latency in their metrics, you’d scale the queriers.
# Example deployment for queriers
apiVersion: apps/v1
kind: Deployment
metadata:
name: loki-querier
namespace: loki
spec:
replicas: 2 # We'll scale this up
selector:
matchLabels:
app: loki-querier
template:
metadata:
labels:
app: loki-querier
spec:
containers:
- name: querier
image: grafana/loki:2.9.0
ports:
- containerPort: 3100
# ... other configuration
To fix slow queries:
# Scale up queriers
kubectl scale deployment loki-querier --replicas=4 -n loki
More querier replicas mean more nodes are available to process incoming queries, reducing the load on any single querier and speeding up response times.
The distributor is less frequently scaled independently, as its load is directly tied to the rate of incoming logs, which the ingesters absorb. However, if you see the distributor itself becoming a bottleneck (e.g., high CPU or network saturation on its pods), you’d scale it similarly.
# Example deployment for distributor
apiVersion: apps/v1
kind: Deployment
metadata:
name: loki-distributor
namespace: loki
spec:
replicas: 1 # Might be scaled up
selector:
matchLabels:
app: loki-distributor
template:
metadata:
labels:
app: loki-distributor
spec:
containers:
- name: distributor
image: grafana/loki:2.9.0
ports:
- containerPort: 3100
# ... other configuration
To scale the distributor:
# Scale up distributor
kubectl scale deployment loki-distributor --replicas=3 -n loki
This distributes the initial ingestion load across more instances, preventing any single distributor from becoming overwhelmed.
The index, whether it’s Cassandra, DynamoDB, or another backend, is scaled according to the best practices of that specific database. For example, with Cassandra, you’d add more nodes to the cluster. With DynamoDB, you’d adjust provisioned throughput or enable auto-scaling. This is critical because slow index lookups directly translate to slow queries.
The mental model here is that Loki is not a monolithic application, but a set of microservices that can be independently resourced. Each service has a distinct role and a distinct scaling profile. You monitor the metrics for each component (CPU, memory, network, request latency, error rates) to identify which part of the system is saturated and then scale that specific component.
One aspect often overlooked is the interaction between the querier and the index. When a query comes in, the querier first consults the index to find which chunks of data (stored in object storage) contain the matching log lines. If the index is slow to respond, the querier effectively stalls. This means that even if your querier pods are idle, query performance can still be poor if the underlying index store is overloaded or under-provisioned. It’s a shared dependency that needs careful monitoring.
The next challenge you’ll face is optimizing your retention policies to manage object storage costs.