Memcached is a distributed memory object caching system. Loki uses Memcached to store query results for a configurable duration, significantly speeding up repeated queries for the same log data.
Let’s see it in action. Imagine you’re frequently querying for errors from a specific microservice over the last hour. Without caching, Loki would have to scan its index and potentially read from object storage for every single query. With Memcached enabled, the first query populates the cache. Subsequent identical queries, within the cache’s Time-To-Live (TTL), hit Memcached directly, bypassing the slower index and storage lookups.
Here’s a simplified view of how it works:
- Query Received: Loki receives a query, for example,
{app="my-service", level="error"} | __line__!="". - Cache Check: Loki first checks if a result for this exact query (including query parameters and time range, adjusted for cache TTL) exists in Memcached.
- Cache Hit: If found, the cached result is returned immediately. This is lightning fast.
- Cache Miss: If not found, Loki proceeds with the normal query execution:
- It queries its index for relevant chunks.
- It fetches chunk data from object storage.
- It processes the data to find matching log lines.
- Cache Population: Once the results are gathered, Loki stores them in Memcached with a specific key derived from the query and a defined TTL.
- Result Returned: The results are then returned to the user.
The primary benefit is reduced latency for repetitive queries. This is particularly valuable in interactive dashboards, alert rule evaluations, and any scenario where the same data is fetched multiple times in quick succession.
To enable Memcached in Loki, you’ll need to configure the caching section in your loki-local.yaml (or equivalent). You’ll need a running Memcached instance accessible from your Loki pods.
caching:
memcached:
addresses:
- "memcached.loki.svc.cluster.local:11211"
- "memcached2.loki.svc.cluster.local:11211"
timeout: "100ms"
max_item_size: "1MB"
ttl: "15m"
In this configuration:
addresses: This is a list of your Memcached server hostnames and ports. Ensure these are reachable by your Loki instances.timeout: The maximum time Loki will wait for a response from Memcached.100msis a common starting point.max_item_size: The maximum size of a single cached item. This prevents large query results from overwhelming Memcached or causing network issues.1MBis a sensible default.ttl: Time-To-Live for cached items.15mmeans results will be kept in Memcached for 15 minutes, after which they will be considered stale and a fresh query will be executed.
The key used for caching is a hash of the query, including the query string, tenant ID, and the effective time range (Loki adjusts the requested time range to align with cache entry boundaries for better hit rates). This ensures that only identical queries with similar time windows hit the cache.
When you’re tuning the ttl, consider your query patterns. If your dashboards typically show data from the last 5 minutes and are refreshed every minute, a ttl of 5m or 10m might be optimal. If you’re running periodic batch analysis over historical data, a longer ttl might be beneficial, but be mindful of potential stale data if the underlying logs change.
A common misconception is that Memcached caches all query results indefinitely. This isn’t true; the ttl is crucial. Furthermore, Loki doesn’t cache the raw log lines themselves, but rather the results of a query execution (i.e., the list of log lines that match the query’s criteria within a given time range). This means that if the underlying data changes or is re-ingested in a way that affects query results, the cache will eventually expire and be repopulated with the latest data.
The next logical step is to explore how Loki’s query frontend and ingesters interact with the cache, and how to monitor cache hit/miss ratios.