Grafana variables are not just for filtering; they are the engine that allows you to transform a static dashboard into a dynamic, interactive data exploration tool.
Let’s see this in action. Imagine you have a fleet of servers, each running the same set of applications, and you want a single dashboard to monitor them all.
Here’s a basic Grafana dashboard configuration showing CPU usage for a single server:
{
"rows": [
{
"panels": [
{
"title": "CPU Usage",
"type": "graph",
"datasource": "prometheus",
"targets": [
{
"expr": "avg by (instance) (rate(node_cpu_seconds_total{mode='idle'}[5m]))",
"legendFormat": "{{instance}}"
}
],
"gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 }
}
]
}
]
}
This works, but you’d need a separate dashboard for each server. Now, let’s introduce a variable. In Grafana, you define variables in the dashboard settings.
Dashboard Settings -> Variables -> Add Variable
Name: server
Type: Query
Data source: prometheus (or your data source)
Query: label_values(node_cpu_seconds_total, instance)
Refresh: On Dashboard Load
Selection Options: Multi-value, Include All option
With this variable defined, we can update the panel’s query to use it:
{
"rows": [
{
"panels": [
{
"title": "CPU Usage",
"type": "graph",
"datasource": "prometheus",
"targets": [
{
"expr": "avg by (instance) (rate(node_cpu_seconds_total{instance=~\"$server\"}[5m]))",
"legendFormat": "{{instance}}"
}
],
"gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 }
}
]
}
]
}
Notice the instance=~\"$server\" in the Prometheus query. The $server is replaced by the selected value(s) from the dropdown. The =~ operator in Prometheus allows for regular expression matching, which is crucial when you enable multi-value or "All" options for your variable.
When you load this dashboard, you’ll see a dropdown at the top, populated with your server instances. Selecting one or more (or "All") will dynamically update the graph to show data only for those selected servers.
This templating capability extends far beyond just selecting a server. You can use variables to:
- Select specific applications:
label_values(my_app_metric, app) - Choose specific environments:
label_values(my_metric, environment) - Filter by region or datacenter:
label_values(my_metric, region) - Dynamically set thresholds: Create a variable of type
Customwith values like10,20,30and use$thresholdin your panel’s alert rules.
The mental model is that Grafana acts as a presentation layer. It takes your dashboard definition, fetches available options for your variables from your data sources, presents those options to the user, and then dynamically rewrites the queries for each panel based on the user’s selections before finally requesting the data from your backend systems.
The true power comes when you chain variables. For example, you could have a region variable, and then a server variable whose query depends on the selected region. The query for the server variable would look something like label_values(node_cpu_seconds_total{region=~"$region"}, instance). This creates a hierarchical drill-down experience.
A common pitfall is forgetting to use the correct regex operator (=~ for Prometheus, for example) when your variable is configured to allow multi-value selections. If you use = and select multiple servers, the query will likely fail because it tries to match a single string like server1,server2 against an exact label value, which won’t exist.
When you define a variable of type Custom, you’re essentially creating a static list of options that are not fetched from a data source. This is incredibly useful for defining fixed sets of values, like different types of analysis or specific reporting periods that aren’t directly available as time-series labels.
The next step is orchestrating multiple panels that all react to the same set of variables, creating a cohesive and responsive monitoring experience.