Fix Grafana "Snapshot Not Found" After Expiry (2026)

Grafana’s snapshotting mechanism failed because the underlying storage backend rejected a write operation, which is usually due to expired credentials or insufficient permissions.

Common causes and their fixes:

Expired S3 Credentials: If your Grafana instance is configured to store snapshots in an S3 bucket, the IAM credentials used by Grafana might have expired.
- Diagnosis: Check your Grafana configuration file (e.g., /etc/grafana/grafana.ini) for S3 settings. Look for access_key and secret_key or a role_arn. If using keys, check their expiry date in AWS IAM. If using a role, check the trust policy and assume-role session duration.
- Fix:
  - If using access/secret keys: Update the access_key and secret_key in grafana.ini with newly generated, non-expired keys from AWS IAM.
  - If using IAM role: In AWS IAM, regenerate the access keys for the user associated with the role or, preferably, update the trust policy of the role to allow Grafana’s EC2 instance profile (or equivalent) to assume it. For temporary credentials obtained via aws sts assume-role, ensure the session duration is sufficient.
  - If Grafana is running on an EC2 instance: Ensure the instance has an IAM role attached with s3:PutObject, s3:GetObject, s3:ListBucket, and s3:DeleteObject permissions for the target snapshot bucket.
- Why it works: Grafana cannot authenticate with S3 to write or retrieve snapshots without valid credentials, leading to the "Snapshot Not Found" error when it tries to access a non-existent or inaccessible snapshot.
Incorrect S3 Bucket Policy: The S3 bucket policy might have been modified, revoking Grafana’s write permissions.
- Diagnosis: Navigate to your S3 bucket in the AWS console. Go to the "Permissions" tab and examine the "Bucket policy." Look for statements that grant s3:PutObject and s3:DeleteObject actions to the principal Grafana is using (e.g., IAM user or role).
- Fix: Add or modify the bucket policy to explicitly grant the necessary permissions. For example:
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::ACCOUNT_ID:user/GRAFANA_USER"
            },
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::your-grafana-snapshot-bucket/*",
                "arn:aws:s3:::your-grafana-snapshot-bucket"
            ]
        }
    ]
}
```
  Replace ACCOUNT_ID, GRAFANA_USER, and your-grafana-snapshot-bucket with your specific values.
- Why it works: A restrictive bucket policy overrides IAM user/role permissions, preventing Grafana from interacting with the bucket even if its IAM identity is valid.
Insufficient IAM Permissions: The IAM user or role assigned to Grafana might lack the necessary permissions to perform S3 operations.
- Diagnosis: In AWS IAM, find the user or role associated with your Grafana instance. Check its attached policies for s3:PutObject, s3:GetObject, s3:ListBucket, and s3:DeleteObject permissions on the target snapshot bucket.
- Fix: Attach a policy that grants these permissions. A managed policy like AmazonS3FullAccess can be used for testing, but a more granular custom policy is recommended for production. Ensure the Resource correctly specifies your snapshot bucket and any objects within it (e.g., arn:aws:s3:::your-grafana-snapshot-bucket/*).
- Why it works: IAM policies define what actions an identity can perform on AWS resources. Without explicit permissions, Grafana is blocked from writing or reading snapshots.
Incorrect S3 Endpoint Configuration: If you’re using a custom S3-compatible storage (like MinIO) or a specific AWS region, the S3 endpoint might be misconfigured.
- Diagnosis: Examine the s3_endpoint setting in your grafana.ini under the [external_image_uploader] section. Verify this endpoint is correct and reachable from your Grafana server.
- Fix: Update s3_endpoint to the correct URL for your S3-compatible storage or the AWS region. For example, for us-east-1, it would be s3.us-east-1.amazonaws.com. If using MinIO, it might be http://minio.example.com:9000. Ensure Grafana can resolve and connect to this endpoint.
- Why it works: Grafana needs to know the precise network address of the S3 service to send its API requests. An incorrect endpoint means Grafana attempts to communicate with the wrong server.
Network Connectivity Issues: Firewalls or network misconfigurations could be blocking Grafana’s outbound connections to S3.
- Diagnosis: From the server running Grafana, attempt to curl the S3 endpoint (e.g., curl -I https://your-grafana-snapshot-bucket.s3.amazonaws.com/). Check network security groups and firewall rules on the Grafana host and any network appliances.
- Fix: Ensure that outbound traffic on port 443 (for HTTPS) is allowed from the Grafana server to the S3 endpoint. If using a proxy, configure Grafana to use it.
- Why it works: Network restrictions can prevent the HTTP/S requests from Grafana to S3 from ever reaching their destination, making the storage appear unavailable.
Snapshot Bucket Not Found or Deleted: The S3 bucket configured for snapshots might have been accidentally deleted or renamed.
- Diagnosis: Verify the existence and exact name of the S3 bucket specified in grafana.ini by checking your AWS S3 console.
- Fix: Recreate the S3 bucket with the exact name specified in grafana.ini or update grafana.ini to match the correct bucket name.
- Why it works: If the target bucket doesn’t exist, S3 will reject any operations targeting it, leading to errors that Grafana reports as "Snapshot Not Found."

After fixing these issues, you might encounter a "502 Bad Gateway" if your Grafana instance’s web server (e.g., Nginx or Apache) is not correctly configured to proxy requests to the Grafana backend application.