Loki can store its index and chunks on object storage like S3 or GCS, but it doesn’t actually use S3 or GCS directly for its primary operations.
Let’s see Loki in action, specifically how it interacts with S3. Imagine we have a Loki instance configured to use S3 for its object store.
# loki-config.yaml
storage:
aws:
s3: s3://my-loki-bucket/
s3_endpoint: "" # Leave empty for AWS regions, or specify e.g. "s3.us-east-1.amazonaws.com"
s3_region: us-east-1
s3_access_key_id: "YOUR_ACCESS_KEY_ID"
s3_secret_access_key: "YOUR_SECRET_ACCESS_KEY"
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: index_
period: 24h
When Loki writes a chunk of data, it doesn’t just dump it into the my-loki-bucket on S3. Instead, it writes it to a local BoltDB file on disk first. Once that BoltDB file reaches a certain size (or after a certain time, depending on configuration), Loki "ships" it to S3. This shipping process involves reading the data from the BoltDB file, potentially compressing it, and then uploading it as a single object to the specified S3 bucket. The index_ prefix ensures all index-related objects are grouped.
Similarly, when you query Loki, it first consults its index (which is also stored in S3, but often a smaller, more frequently accessed version might be cached locally or in memory). The index tells Loki which S3 objects (chunks) contain the data for your query. Loki then downloads the relevant chunk objects from S3, decompresses them if necessary, and reads the data to return your query results.
The core problem Loki’s object storage backends solve is decoupling long-term storage from the Loki process itself. This means Loki instances can be ephemeral. You can scale them up and down, restart them, or even replace them entirely, and your data remains safe and accessible in S3 or GCS. It also allows for virtually infinite scalability of your log storage without needing to manage complex disk arrays for your Loki instances. The boltdb-shipper is crucial here; it’s the mechanism that bridges the gap between Loki’s immediate operational needs and the long-term, durable storage provided by object stores.
The schema_config defines how Loki structures its data. store: boltdb-shipper tells Loki to use BoltDB on disk and then ship it. object_store: s3 specifies S3 as the destination for those shipped files. The schema: v11 indicates the version of Loki’s internal data schema, which evolves over time and affects how data is indexed and stored. index.prefix and index.period control how the index files are organized within the object store.
The actual interaction with S3 or GCS is handled by the underlying object store client library within Loki. For S3, this means using the AWS SDK, and for GCS, it’s the Google Cloud Storage client library. When s3_endpoint is empty, Loki defaults to using the standard AWS endpoints for the specified s3_region. If you’re using a compatible object storage service (like MinIO or Ceph) or a different cloud provider’s object storage that mimics the S3 API, you’d specify its endpoint here. Credentials are provided via s3_access_key_id and s3_secret_access_key (or environment variables/IAM roles, which are more secure in production).
When Loki needs to retrieve data, it constructs the object key based on the index and chunk information. For example, a chunk might be stored as index_12345/67890.gz. Loki then makes an HTTP GET request to the S3 endpoint for that specific object. The boltdb-shipper ensures that all data belonging to a particular BoltDB file is uploaded as a single object, which optimizes retrieval as Loki only needs to download one file per shipped BoltDB.
If you’re using GCS, the configuration would look very similar, but you’d use storage.google.cloud_storage instead of storage.aws, and provide bucket_name, project_id, and authentication details (like a json_credentials_file path).
The most surprising thing about Loki’s object storage is that it doesn’t store individual log lines or small log files directly. Instead, it aggregates logs into larger chunks, writes them to BoltDB locally, and then ships these BoltDB files as single objects to object storage. This chunking and shipping strategy is key to its performance and scalability, as it minimizes the number of individual operations against the object store.
The next hurdle you’ll likely face is understanding Loki’s index store, which is a separate configuration from the object store.