Grafana dashboards are often static snapshots of system state, but they become infinitely more useful when you can overlay transient events like deployments.

Imagine you’re staring at a spike in latency on a dashboard, and you don’t know why. Was it a bad deploy? A traffic surge? A database issue? Without context, it’s a guessing game. Annotations are Grafana’s way of telling that story by marking significant events directly on your graphs.

Let’s say you’ve just deployed a new version of your user-service. You want to see if any metrics changed noticeably around the deployment time.

Here’s a simplified user-service deployment script snippet using kubectl:

# Build the Docker image
docker build -t my-registry/user-service:v1.2.0 .

# Push the image
docker push my-registry/user-service:v1.2.0

# Apply the Kubernetes deployment
kubectl apply -f kubernetes/deployment.yaml

Now, how do we get that deployment event into Grafana? We need a way to programmatically tell Grafana "hey, a deployment happened here." The Grafana API is the key.

The Grafana HTTP API has an endpoint for creating annotations: POST /api/annotations.

Here’s a curl command that would create an annotation for our deployment:

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_GRAFANA_API_KEY" \
  -d '{
    "dashboardId": 123,
    "panelId": 456,
    "time": 1678886400000,
    "isRegion": false,
    "tags": ["deployment", "user-service", "v1.2.0"],
    "text": "User Service deployed v1.2.0"
  }' \
  http://your-grafana-host:3000/api/annotations

Let’s break this down:

  • dashboardId: 123: The ID of the Grafana dashboard you want to annotate. You can find this in the dashboard URL.
  • panelId: 456: The ID of the specific panel on the dashboard. Also in the URL when viewing the panel.
  • time: 1678886400000: The Unix timestamp in milliseconds when the event occurred. This is crucial for aligning annotations with graph data. You’d get this from your CI/CD system.
  • isRegion: false: Indicates this is a point-in-time event, not a duration.
  • tags: Useful for filtering annotations. Here, we’re tagging it with deployment, the service name, and the version.
  • text: A human-readable description of the event.

The most common way to automate this is within your CI/CD pipeline. After your deployment successfully completes, your pipeline script would trigger this curl command. Tools like Jenkins, GitLab CI, GitHub Actions, or Argo CD can all execute shell commands.

Many CI/CD systems also have dedicated Grafana plugins or integrations that simplify this. For instance, if you’re using GitLab CI, you might have a grafana_annotation job that runs after deployment.

Here’s how you’d integrate this into a hypothetical GitLab CI .gitlab-ci.yml:

stages:
  - deploy
  - notify

deploy_user_service:
  stage: deploy
  script:
    - echo "Building and deploying user-service..."
    # ... (your docker build, push, kubectl apply commands) ...
    - export DEPLOY_TIMESTAMP=$(date +%s%3N) # Get timestamp in milliseconds
  only:
    - main

notify_grafana:
  stage: notify
  script:
    - |
      curl -X POST \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer ${GRAFANA_API_TOKEN}" \
        -d '{
          "dashboardId": 123,
          "panelId": 456,
          "time": '"${DEPLOY_TIMESTAMP}"',
          "tags": ["deployment", "user-service", "v1.2.0"],
          "text": "User Service deployed v1.2.0"
        }' \
        http://your-grafana-host:3000/api/annotations
  variables:
    GRAFANA_API_TOKEN: YOUR_GRAFANA_API_TOKEN_FROM_CI_VARS
  needs:
    - deploy_user_service
  only:
    - main

Notice the DEPLOY_TIMESTAMP variable being captured and used. The GRAFANA_API_TOKEN should be stored as a protected CI/CD variable.

When you view your Grafana dashboard, you’ll see a small vertical line or marker on the graph at the time of deployment, with the text "User Service deployed v1.2.0" appearing if you hover over it. If you have many annotations, you can filter them using the tags you defined.

The text field can be much more elaborate. You can include links to your CI/CD job, commit SHA, or even rollback instructions. For example:

{
  "dashboardId": 123,
  "panelId": 456,
  "time": 1678886400000,
  "tags": ["deployment", "user-service", "v1.2.0"],
  "text": "User Service v1.2.0 deployed successfully.\nCommit: <a href=\"https://github.com/myorg/user-service/commit/abc1234\">abc1234</a>\nCI Job: <a href=\"https://gitlab.com/myorg/user-service/-/jobs/567890\">#567890</a>"
}

This uses HTML within the text field, which Grafana renders.

A common pitfall is not having the correct Grafana API key or not having the Editor role (or higher) for the user associated with the API key. The API key must have permissions to create annotations. Also, ensure your dashboardId and panelId are accurate; a mismatch will result in the annotation not appearing on the intended graph.

The actual timestamp for the annotation is determined by the time field. If this field is missing or malformed, Grafana might default to the current time or reject the request.

Once you have annotations set up for deployments, the next logical step is to automate alerts based on metric deviations around those annotation events.

Want structured learning?

Take the full Grafana course →