The most surprising thing about Grafana time series visualizations is how much they can lie to you if you’re not careful.
Here’s a look at a typical Grafana dashboard panel showing CPU utilization.
{
"datasource": "Prometheus",
"targets": [
{
"expr": "avg(node_cpu_seconds_total{mode!=\"idle\"}) by (instance)",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"title": "CPU Utilization",
"type": "timeseries"
}
This panel, at first glance, seems straightforward: it shows CPU usage per server instance. But what if your CPU utilization suddenly spikes to 95% across all your instances, and then immediately drops back to 10%? This could be a genuine, brief overload, or it could be a sampling artifact. Grafana, by default, connects the dots between data points. If your scrape interval is 15 seconds, and a spike happens between two scrapes, you won’t see it. The graph will just show a smooth transition from 10% to 10%, making it look like nothing happened.
To truly understand what’s happening, you need to consider the nature of the data you’re collecting and how Grafana interpolates it. The timeseries panel type, by default, uses linear interpolation. This means that between any two data points, Grafana draws a straight line. If your data is collected at discrete intervals (like Prometheus scrapes), this linear interpolation can smooth over transient events that occur between those intervals.
Consider this scenario: a process on a server briefly consumes 100% CPU for 1 second, then returns to normal. If your Prometheus scrape interval is 15 seconds, that 1-second spike will likely be missed entirely by the scrape. The data points collected will be before and after the spike, and Grafana’s linear interpolation will simply draw a line connecting those two points, masking the event.
The null value and null point options in the panel settings are crucial here. If your data source can return null values (e.g., if a metric is temporarily unavailable), Grafana has options for how to display these. null as zero will treat missing data points as 0, which can be misleading. connected (the default for linear interpolation) draws a line. null will show a gap. For high-frequency, transient events, understanding how your data is sampled and how Grafana connects these samples is paramount. If you’re looking for short-lived, high-intensity events, you might need to adjust your data collection frequency or use a visualization option that highlights gaps or sudden changes.
To get a more accurate picture of rapid changes, especially when dealing with high-frequency data, you can experiment with different null value and null point settings. For example, setting null value to null will create visible gaps in your graph where data points are missing, which can help identify sampling issues or brief metric outages. The transform option, specifically reduce, can also be useful. By using last or first in a reduce transformation, you can effectively choose which data point Grafana uses if multiple are available within a time bucket, giving you more control over how aggregation happens.
The legend field in a Grafana target is not just for display; it’s a powerful templating mechanism. Using {{instance}} as shown above allows you to dynamically label each series with the server name it came from. This is essential for distinguishing between different sources of data on the same graph. You can also use {{job}} or other Prometheus labels, or even custom variables defined in your Grafana dashboard, to create highly dynamic and informative legends.
One common pitfall is not understanding the difference between count and sum when aggregating metrics over time. If you’re graphing rate(http_requests_total[5m]), you’re looking at the average rate of requests per second over the last 5 minutes. If you were to instead try to sum this rate, you’d get a nonsensical result. The rate function in Prometheus already handles the aggregation over the specified window. When creating your own aggregations in Grafana or within your data source queries, be mindful of what each aggregation function truly represents and how it interacts with the underlying metric.
The tooltip display options are also key. By default, Grafana shows the values for all series at a given point in time. However, you can customize this to show only specific series, or even to display aggregated values across multiple series. This can help declutter your tooltip and focus on the most relevant information when you hover over a graph.
When you start using Grafana transformations like Filter data by values or Group by, you’re essentially pre-processing the data before it’s rendered. This can simplify complex queries or combine data from multiple sources in a way that’s easier to visualize. For instance, you might filter out specific error codes from a log stream before graphing the overall request rate.
The next concept you’ll likely grapple with is how to effectively use Grafana’s alerting features to proactively notify you about these transient events, rather than just spotting them after the fact.