Istio doesn’t collect metrics, logs, and traces; it configures other services to do it for them.

Let’s see Istio’s telemetry API in action. Imagine we have a simple frontend service talking to a backend service. We want to ensure every request to backend is logged, and we want detailed traces for requests that take longer than 500ms.

Here’s the frontend deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: docker.io/istio/examples-bookinfo-frontend:1.16.0
        ports:
        - containerPort: 8080

And the backend:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
      - name: backend
        image: docker.io/istio/examples-bookinfo-backend:1.16.0
        ports:
        - containerPort: 8080

Now, let’s apply Istio’s telemetry configuration. We’ll use Telemetry resources to define our desired observability.

First, let’s enable access logs for all traffic going to the backend service. We’ll create a Telemetry resource targeting the backend workload:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: backend-access-logs
  namespace: default # Assuming your services are in the 'default' namespace
spec:
  selector:
    matchLabels:
      app: backend
  accessLogs:
  - providers:
    - name: istiod # This is the default provider name for logs

When Istio’s control plane (specifically istiod) processes this Telemetry resource, it generates configuration for the Envoy sidecars injected into the backend pods. For each incoming request to the backend service, Envoy will now emit an access log. These logs typically go to standard output by default within the sidecar container, which Kubernetes then aggregates. You can view these logs using kubectl logs <backend-pod-name> -c istio-proxy.

Next, let’s configure distributed tracing. We want to trace requests to backend but only if they exceed a certain latency threshold. This is a powerful capability: we can selectively sample traces based on request characteristics.

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: backend-tracing
  namespace: default
spec:
  selector:
    matchLabels:
      app: backend
  tracing:
  - providers:
    - name: opentelemetry # Or your configured tracing provider name
      spanName: backend-service
      sampling: 100.0 # Default sampling, we'll refine this
    match:
    - name: latency
      value: ">500ms" # Only trace if latency is greater than 500ms

This Telemetry resource instructs the Envoy sidecar on the backend to:

  1. Generate Spans: For every request it handles, it will create a span representing its work.
  2. Propagate Traces: It will forward any incoming trace context (like x-b3-traceid) or inject a new one if none exists.
  3. Sample Conditionally: The match field is key here. Envoy will only send tracing information for spans where the operation’s duration (latency) exceeds 500 milliseconds. This is a form of tail-based sampling, but configured at the edge proxy.
  4. Send to Provider: The generated spans are sent to the configured tracing backend (e.g., Jaeger, Zipkin, OpenTelemetry Collector). The provider name here should match what you’ve configured in your Istio installation for tracing.

The sampling: 100.0 in the providers block indicates that 100% of the requests that meet the match criteria will be traced. If we had removed the match block, this would mean 100% of all requests would be traced, which can be very noisy and expensive.

To see this in action, you’d typically have a tracing backend deployed (like Jaeger) and a way to generate traffic. After sending some requests to frontend, which in turn calls backend, you would query your tracing backend. You’d see logs from the backend sidecar and traces for the slower requests to backend.

The most surprising true thing about Istio’s telemetry configuration is that it doesn’t own the observability pipeline; it merely orchestrates it by pushing configuration to the Envoy sidecars. Envoy acts as the data plane agent, collecting and exporting telemetry based on the rules defined in Istio’s Telemetry custom resources. This decoupling means you can swap out your backend observability tools (e.g., from Jaeger to Datadog) without changing your Istio Telemetry configurations, as long as the new backend can consume the telemetry format Envoy exports (like OTLP, Jaeger Thrift, etc.).

The Telemetry API is a powerful abstraction, but understanding how it translates into Envoy configuration is crucial for effective debugging and fine-tuning. You’re not just saying "log this"; you’re telling Envoy how to log it, where to send it, and under what conditions to do so.

The next step you’ll likely encounter is configuring custom metrics based on request attributes or response codes.

Want structured learning?

Take the full Istio course →