Loki’s ring is a sophisticated, distributed hash table that underpins its high availability and scalability by ensuring data is evenly distributed across all ingesters and queriers.
Let’s see this in action. Imagine we have a small Loki cluster with three ingester instances: ingester-1, ingester-2, and ingester-3. When a log line arrives, Loki needs to decide which ingester is responsible for storing it. This decision is made by hashing the log stream’s unique identifier (usually a combination of labels like app, namespace, and pod) and then using the ring to find the responsible ingester.
Here’s a simplified view of what that might look like internally. The ring is essentially a list of "tokens" (large integers) sorted numerically. Each ingester instance owns a range of these tokens. When a log stream’s hash falls within a specific token range, the ingester responsible for that range is selected.
Ring State (Simplified):
Tokens:
1000 (owned by ingester-1)
5000 (owned by ingester-2)
9000 (owned by ingester-3)
Log Stream Hash: 3500
Ring Lookup:
3500 falls between 1000 and 5000.
Therefore, ingester-2 is responsible for this log stream.
This mechanism is called consistent hashing, and its "consistent" part is key. If ingester-2 goes down, the ring doesn’t completely rebalance. Instead, the tokens previously owned by ingester-2 are redistributed to its next nearest neighbors in the ring. This minimizes data movement and ensures that most of the data remains with its original ingester, making the system resilient to failures.
The ring is managed by the Loki memberlist component, which uses gossip to keep all instances updated on the state of the ring. When an ingester joins or leaves the cluster, this information is propagated, and the ring is dynamically updated.
The primary problem Loki’s ring solves is the uniform distribution of load and data. Without it, you’d need a central coordinator to decide where to send logs, creating a bottleneck. Or, you’d have to rebalance everything manually whenever an instance changed, which is impractical at scale. The ring allows Loki to scale horizontally by simply adding more ingesters or queriers, and the ring automatically adjusts to distribute the load.
The configuration for the ring is typically found in your Loki loki-config.yaml file, under the ingester and querier sections. You’ll see parameters like join_multicast_group and join_port which are used by the memberlist to discover other Loki instances.
ingester:
join_multicast_group: "239.0.0.1"
join_port: 7946
# ... other ingester settings
querier:
join_multicast_group: "239.0.0.1"
join_port: 7946
# ... other querier settings
This setup allows ingesters and queriers to form a gossip cluster, and within that cluster, the ring is built and maintained. The num_tokens parameter (often set to 256 by default) controls the granularity of the ring. A higher number of tokens means more precise distribution but also a slightly larger ring state to manage.
The magic of the ring’s dynamic rebalancing lies in the fact that each ingester instance, when it starts, announces its presence and the token ranges it intends to own. Other instances in the memberlist gossip cluster hear this announcement and update their local view of the ring. If an instance stops announcing, its tokens are considered "lost" or "unowned," and the ring rebalancing process kicks in to assign those tokens to other available instances. This happens automatically and continuously.
A common point of confusion is understanding how many replicas Loki uses for a given piece of data. Loki doesn’t inherently replicate data across multiple ingesters at the storage level in the same way a traditional distributed database might. Instead, the ring ensures that for any given log stream, there are multiple potential ingesters that could be responsible for it at any given moment. If the primary ingester for a stream becomes unavailable, the ring will eventually route that stream’s logs to a different ingester, which will then be responsible for storing it. The actual replication strategy for the underlying object storage (like S3 or GCS) is handled by that storage system itself.
The next concept you’ll likely encounter is how queriers use this same ring to efficiently locate the data they need to serve queries.