Linkerd’s "golden signals" aren’t just metrics; they’re the heartbeat of your distributed system, revealing its health and performance in real-time.
Let’s watch Linkerd in action. Imagine a simple microservice architecture: a frontend service calling an api service, which in turn calls a database service.
# frontend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
spec:
replicas: 2
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: app
image: your-docker-repo/frontend-service:v1.0
ports:
- containerPort: 8080
---
# api-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 3
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: app
image: your-docker-repo/api-service:v1.2
ports:
- containerPort: 5000
---
# database-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: database
spec:
replicas: 1
selector:
matchLabels:
app: database
template:
metadata:
labels:
app: database
spec:
containers:
- name: app
image: your-docker-repo/database-service:v1.0
ports:
- containerPort: 5432
When you install Linkerd (e.g., linkerd install | kubectl apply -f -), it injects a linkerd-proxy container into each of your pods. This proxy sits alongside your application container, transparently handling all inbound and outbound network traffic.
# Example pod with Linkerd proxy injected
kubectl get pod frontend-abcde-fghij -o yaml
# ...
spec:
containers:
- name: app
image: your-docker-repo/frontend-service:v1.0
# ...
- name: linkerd-proxy
image: linkerd.io/proxy:v2.13.0 # Example version
ports:
- containerPort: 4140 # Proxy's outbound port
args:
- /proxy
- --outgoing-addr=:4140
# ... other proxy configurations
Now, let’s look at the golden signals themselves, using linkerd viz commands.
Traffic
This is the most basic signal: how much is happening? Linkerd tracks the success and total number of requests between services.
# See traffic from frontend to api
linkerd viz tap -n default frontend --to api
# Output might look like:
# SRC DST FLIGHT R/S P50 P90 P99 R%
# frontend api 100 100.00 10ms 25ms 50ms 99.99%
Here, R/S (Requests per Second) and R% (Success Rate) are key. A drop in R/S could mean your frontend is slowing down or failing to send requests. A low R% on the api side indicates the api is returning errors (e.g., HTTP 5xx).
Latency
Latency tells you how fast your requests are being served. Linkerd provides percentiles (P50, P90, P99) for request durations.
# See latency for requests to the api service
linkerd viz stat --to api -n default
# Output might look like:
# R/S SUCCESS R% EMSP-P50 EMSP-P90 EMSP-P99 RS-P50 RS-P90 RS-P99 MES-P50 MES-P90 MES-P99
# 100 100 100% 10ms 25ms 50ms 2ms 5ms 10ms 12ms 30ms 60ms
EMSP (End-to-Middle Service Processing) latency is the total time from when the request left the source service until it was fully processed by the destination service. RS (Request Start) is the time from when the request arrived at the destination service’s proxy until it was handed off to the application. MES (Middle-to-End Service) is the time spent within the application itself. High EMSP could be your api service, high RS could be the api’s proxy, and high MES is definitely your application code.
Errors
This is the most critical signal: are requests failing? Linkerd categorizes errors into successful requests, client-side errors (HTTP 4xx), and server-side errors (HTTP 5xx).
# See error rates for requests from frontend to api
linkerd viz stat --to api -n default
# Output will include columns like:
# SUCCESS R% CLIENT_ERR CLIENT_ERR% SERVER_ERR SERVER_ERR%
# 100 100% 0 0% 0 0%
A rising SERVER_ERR% (e.g., 5xx errors) means the api service is failing. A rising CLIENT_ERR% (e.g., 4xx errors) means the frontend is sending requests that the api service considers invalid, or a client-side timeout occurred.
The magic here is that Linkerd’s proxy is observing these requests transparently. It doesn’t need application code changes. It measures the time between when it sends a request and when it receives a response, and it inspects the response’s status code.
These signals are interconnected. A sudden spike in latency often precedes a spike in errors. A drop in traffic might be a symptom of upstream failures.
The Linkerd proxy’s linkerd-proxy container is the unsung hero. It’s a high-performance Rust binary that sits in each pod. It intercepts all TCP connections made by your application container and all incoming connections. For every outbound request, it injects a small l5d-request-id header and measures the time and success of the response. For inbound requests, it records the same. This data is then aggregated and exposed via the Prometheus endpoint (/metrics) on the proxy’s default port, 4140.
The real power comes when you combine these signals. For instance, you can create Grafana dashboards showing the Golden Signals alongside your application’s specific metrics. If you see a P99 latency spike on the api service, you can then drill down into the api’s logs or application-specific metrics to understand why that particular request was slow.
The linkerd-proxy’s configuration, particularly the policy.linkerd.io Kubernetes Custom Resource, allows you to define fine-grained access control and traffic policies. While this isn’t directly a golden signal, it influences traffic flow and can indirectly impact latency and errors if misconfigured, for example, by blocking legitimate traffic or imposing unnecessary routing delays.
The next step is understanding how to use these signals for proactive alerting, moving beyond just observation to automated incident response.