Minikube, Prometheus, and Grafana, when deployed together, form a powerful local monitoring stack, but the most surprising truth is that your local development environment can become more complex and resource-intensive than many production deployments.
Let’s see this stack in action. Imagine you’ve just spun up Minikube and deployed Prometheus and Grafana using Helm. Your kubectl get pods -n monitoring might look something like this:
NAME READY STATUS RESTARTS AGE
prometheus-grafana-0 2/2 Running 0 5m
prometheus-kube-state-metrics-0 1/1 Running 0 5m
prometheus-node-exporter-lqg4b 1/1 Running 0 5m
prometheus-prometheus-oper-prometheus-0 2/2 Running 0 5m
Here, prometheus-grafana-0 is your Grafana instance, prometheus-node-exporter-0 is collecting host metrics, prometheus-kube-state-metrics-0 is exposing Kubernetes object state as metrics, and prometheus-prometheus-oper-prometheus-0 is your Prometheus server.
The core problem this stack solves is the lack of visibility into your local Kubernetes cluster. Without it, debugging application issues, understanding resource utilization, or simply knowing if your deployments are healthy becomes a guessing game. Prometheus acts as the data collector and time-series database, scraping metrics from various exporters (like node-exporter for host-level metrics and kube-state-metrics for cluster-level object states) and your applications if they expose Prometheus-compatible endpoints. Grafana then queries Prometheus to visualize this data in dashboards.
The mental model for this stack is a data pipeline. Exporters are the data sources, Prometheus is the central ingestion point and storage, and Grafana is the presentation layer.
Here’s a typical Helm values.yaml for deploying Prometheus and Grafana to Minikube:
grafana:
enabled: true
adminPassword: "changeme"
persistence:
enabled: true
size: 1Gi
ingress:
enabled: true
hosts:
- grafana.minikube.local
prometheus:
prometheusSpec:
serviceMonitorSelector: {} # Scrape all services
podMonitorSelector: {} # Scrape all pods
storageSpec:
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 2Gi
retention: 24h
resources:
requests:
cpu: "200m"
memory: "500Mi"
limits:
cpu: "500m"
memory: "1Gi"
# kube-state-metrics and node-exporter are typically enabled by default
# but you can configure them here if needed.
To access Grafana, you’d typically port-forward: kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80. Then, navigate to http://localhost:3000 and log in with admin and the password you set (changeme in this example). You’ll need to configure Prometheus as a data source in Grafana, usually at http://prometheus-operated.monitoring.svc.cluster.local:9090.
A key lever you control is Prometheus’s scraping configuration. By default, the Prometheus Operator (which Helm usually installs) is configured to discover and scrape targets automatically using ServiceMonitor and PodMonitor custom resources. If your application isn’t showing up, ensure it has the correct labels that match the operator’s selectors, and that your application’s metrics endpoint is exposed. For example, a ServiceMonitor might look like this:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: my-app-monitor
namespace: default
spec:
selector:
matchLabels:
app: my-app # Label on your application's Service
namespaceSelector:
matchNames:
- default
endpoints:
- port: metrics
interval: 30s
The most overlooked aspect of this local setup is the resource allocation. Minikube, by default, might only allocate 2 CPUs and 4GB of RAM. Prometheus, especially with extensive scraping and retention, can easily consume a significant portion of this. If your Prometheus pod restarts with OOMKilled (Out Of Memory) or your Minikube VM becomes unresponsive, the first place to check is the resource requests and limits for Prometheus in your Helm values.yaml and the overall resources allocated to Minikube itself (e.g., minikube start --cpus 4 --memory 8192). The storageSpec for Prometheus is also critical; a small persistent volume will lead to Prometheus discarding old data too quickly or failing to store new data if it fills up.
Once you have basic dashboards working, the next natural step is to explore pre-built Grafana dashboards for Kubernetes, often found on Grafana Labs’ official dashboard repository, or to start crafting custom dashboards for your specific applications.