Prometheus metrics for Memcached are not just a window into your cache’s performance; they’re a crucial part of understanding how your distributed systems are actually behaving.
Let’s see it in action. Imagine you’ve got a Memcached instance running on 192.168.1.100:11211. We want to expose its metrics so Prometheus can scrape them.
First, you need a Prometheus exporter. The memcached_exporter is the de facto standard. You’d typically run this as a separate service. Here’s a basic Docker command to get it going:
docker run -d \
--name memcached_exporter \
-p 9150:9150 \
quay.io/prometheus/memcached-exporter:latest \
--memcached.address="tcp://192.168.1.100:11211"
This starts the exporter on port 9150 and tells it to connect to your Memcached instance. Prometheus then needs to be configured to scrape this exporter. In your prometheus.yml file, you’d add a job like this:
scrape_configs:
- job_name: 'memcached'
static_configs:
- targets: ['localhost:9150'] # If running exporter on the same host as Prometheus
Or, if the exporter is running elsewhere:
scrape_configs:
- job_name: 'memcached'
static_configs:
- targets: ['192.168.1.200:9150'] # Replace with exporter's IP and port
Once Prometheus is scraping, you’ll see metrics like memcached_items, memcached_get_hits, memcached_get_misses, and memcached_evictions appearing in your Prometheus UI. These aren’t just raw numbers; they tell a story. memcached_get_hits and memcached_get_misses directly inform your cache hit ratio, a fundamental measure of cache effectiveness. High evictions (memcached_evictions) indicate your cache is full and struggling to keep up with the demand, forcing out valuable data.
The memcached_exporter works by connecting to the Memcached instance via its stats command. This command, when issued to Memcached, returns a wealth of internal statistics. The exporter parses these statistics and exposes them as Prometheus metrics, transforming raw Memcached output into a format Prometheus can understand and query. It’s a simple but elegant proxy.
Now, let’s talk about alerting. A common and critical alert is for a low cache hit ratio. You can define a Prometheus alert rule like this:
groups:
- name: memcached.rules
rules:
- alert: HighMemcachedMissRatio
expr: |
sum(rate(memcached_get_misses[5m])) by (instance)
/
(
sum(rate(memcached_get_hits[5m])) by (instance)
+
sum(rate(memcached_get_misses[5m])) by (instance)
)
* 100 > 75
for: 5m
labels:
severity: warning
annotations:
summary: "Memcached instance {{ $labels.instance }} has a high miss ratio ({{ $value | printf "%.2f" }}%)"
description: "The Memcached instance {{ $labels.instance }} is experiencing a cache miss ratio above 75% for the last 5 minutes. This could indicate insufficient cache capacity or inefficient caching strategies."
This rule calculates the miss ratio over a 5-minute window. If it exceeds 75% for more than 5 minutes, it fires a warning. The rate() function is key here; it calculates the per-second average rate of increase of the counters, giving you a real-time performance snapshot rather than just cumulative counts. The sum by (instance) ensures the calculation is done per Memcached instance being scraped.
Another essential alert is for excessive evictions:
groups:
- name: memcached.rules
rules:
- alert: HighMemcachedEvictions
expr: sum(rate(memcached_evictions[5m])) by (instance) > 0
for: 10m
labels:
severity: critical
annotations:
summary: "Memcached instance {{ $labels.instance }} is experiencing evictions."
description: "The Memcached instance {{ $labels.instance }} has had more than 0 evictions per second over the last 10 minutes. This suggests the cache is full and data is being prematurely removed."
This alert triggers if there are any evictions happening per second for a sustained period. While a few evictions might be normal during traffic spikes, a continuous stream indicates your cache is too small for your workload.
A subtle point about Memcached’s stats command is that it provides a snapshot. The memcached_exporter polls this regularly. If your Memcached instance is under extreme load, it might be slow to respond to the stats command, or the data might be slightly stale by the time the exporter picks it up. For most use cases, this is perfectly acceptable, but for extremely high-throughput scenarios, you might consider running multiple exporters or tuning the scrape interval in Prometheus.
The next step after monitoring hit ratios and evictions is to correlate these metrics with application-level performance. You’ll want to see if a drop in cache hit ratio directly corresponds to an increase in database load or application response times.