Grafana alert contact points are how your alerts actually reach people, and they’re way more flexible than you probably think.
Let’s see one in action. Imagine we have a critical service, and if its request latency goes above 500ms for more than 5 minutes, we want to know now.
Here’s how you’d set up a contact point to hit both Slack and PagerDuty simultaneously:
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
name: example-grafana
spec:
# ... other grafana config ...
notifiers:
- name: slack-pagerduty-notifier
type: slack
settings:
recipient: "#alerts-channel"
token: "xoxb-your-slack-bot-token"
username: "Grafana Bot"
# This is key: a nested notifier
notifiers:
- name: pagerduty-notifier
type: pagerduty
settings:
routing_key: "your-pagerduty-integration-key"
severity: "critical"
client: "Grafana"
client_url: "http://grafana.example.com"
This grafana.integreatly.org/v1beta1 custom resource defines a Grafana instance. The notifiers section is where the magic happens. We’re defining a primary notifier of type: slack. Inside its settings, we give it a recipient (the Slack channel), an authentication token, and a username for the bot.
But look closely: the Slack notifier itself has a nested notifiers section. This is Grafana’s way of chaining contact points. The nested notifier is of type: pagerduty. Its settings include a routing_key (your PagerDuty integration key), a severity level, and some identifying client information.
When an alert fires and is routed to this combined contact point, Grafana will first send a message to Slack. Then, it will send an event to PagerDuty using the provided integration key. This way, you get immediate visibility in Slack and a guaranteed escalation path through PagerDuty if needed.
The actual alert rule that uses this contact point might look something like this:
apiVersion: monitoring.grafana.org/v1alpha1
kind: AlertRule
metadata:
name: high-request-latency
spec:
title: High Request Latency
condition: "avg(http_requests_total{status_code=~'5..'}) by (job) > 100"
data:
- refId: A
datasourceUid: prometheus-datasource-uid
queryType: range
model:
expr: "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service='my-app'}[5m])) by (le, service)) > 0.5"
intervalMs: 1000
maxDataPoints: 43200
legendFormat: "{{service}}"
refId: A
evaluateFor: 5m
labels:
severity: critical
annotations:
summary: "High request latency for {{ $labels.service }}"
description: "99th percentile latency for {{ $labels.service }} is {{ $value | humanize }}s for the last 5 minutes."
# This is where you link to the contact point
receivers:
- grafana-managed-contact-point: slack-pagerduty-notifier
Here, the AlertRule specifies a condition that checks for a high rate of 5xx errors, but the data section uses a histogram_quantile to look at the 99th percentile latency. It evaluateFor: 5m, meaning the alert only triggers if the condition is true for five minutes straight. Crucially, the receivers field points to the grafana-managed-contact-point we defined earlier by its name, slack-pagerduty-notifier.
The most surprising thing about Grafana’s notification system is its ability to nest contact points arbitrarily, allowing for complex routing and fallback logic directly within the Grafana UI or its configuration. You can have a primary Slack notification, and if that doesn’t get acknowledged within a certain time, it can trigger a PagerDuty incident. Or, you could have multiple email addresses and a webhook all hit simultaneously from a single alert.
The token for Slack is your bot token, which needs to be created in the Slack API dashboard and granted the chat:write scope. For PagerDuty, the routing_key is specific to the integration you set up in PagerDuty (typically under Services -> your service -> Integrations). This key is what PagerDuty uses to identify which integration to trigger. If you’re using the Grafana Operator, these values would typically be managed via Kubernetes secrets.
If you only specify recipient and token for Slack, and routing_key for PagerDuty, you’re good to go. You can also add username and icon_emoji for Slack to customize the bot’s appearance. For PagerDuty, severity can be critical, error, warning, or info, and client_url is useful for providing a direct link back to Grafana for context.
Once this is configured and your alert rule is referencing it, you’ll see messages appearing in your #alerts-channel in Slack and PagerDuty incidents being created when the alert condition is met.
The next thing you’ll likely want to explore is templating alert messages using Go templating within the annotations of your alert rules to dynamically include more context from your metrics.