Adding mutual TLS (mTLS) and observability to your microservices with Istio isn’t just about security and monitoring; it’s about fundamentally changing how your services interact and how you understand their behavior.
Let’s see it in action. Imagine you have two services, frontend and backend.
Initial State (No mTLS, Basic Observability)
# Istio configuration to enable basic ingress and routing
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: frontend-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: frontend
spec:
hosts:
- frontend
gateways:
- frontend-gateway
http:
- route:
- destination:
host: frontend
port:
number: 8080
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: backend
spec:
hosts:
- backend
http:
- route:
- destination:
host: backend
port:
number: 9090
A request comes into the frontend via the ingressgateway. The frontend then calls the backend over plain HTTP. You can see these requests in Prometheus and Grafana, but you can’t be sure who called whom or if the connection was tampered with.
Enabling mTLS
Istio’s mTLS ensures that services only communicate with other services they’re authorized to talk to, and that the communication is encrypted. This is enforced by the Istio sidecar proxy injected into each pod.
First, we need to enable strict mTLS for the entire mesh.
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system
spec:
mtls:
mode: STRICT
This PeerAuthentication policy, applied to the istio-system namespace (which affects the entire mesh by default), tells all Istio sidecars to reject any incoming connections that aren’t TLS, and to present their own client certificates for outgoing connections.
Next, we need to tell the backend service that it’s allowed to receive traffic from the frontend (and any other services that have valid Istio certificates).
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: allow-frontend-to-backend
namespace: default # Assuming backend is in the default namespace
spec:
selector:
matchLabels:
app: backend # Label on your backend pods
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/frontend"] # Service account of the frontend
This AuthorizationPolicy ensures that only the frontend’s service account (identified by its Kubernetes Service Account and the Istio cluster.local identity) can communicate with the backend.
Now, when the frontend calls the backend, the Istio sidecars will:
- The
frontendsidecar will initiate a TLS connection to thebackendsidecar. - The
backendsidecar will present its identity certificate. - The
frontendsidecar will verify thebackend’s certificate against the Istio CA. - The
backendsidecar will verify thefrontend’s certificate against the Istio CA and check if it’s allowed by theAuthorizationPolicy. - If all checks pass, the encrypted tunnel is established.
Enabling Observability
Istio automatically generates telemetry for all traffic flowing through its sidecars. This includes metrics, distributed tracing, and access logs.
Metrics: Istio’s mixer (or telemetry component in newer versions) collects metrics like request volume, latency, and success rates. These are scraped by Prometheus. You can query these metrics to see traffic patterns, identify slow services, and detect errors.
For example, to see the request rate from frontend to backend:
curl 'http://prometheus-server:9090/api/v1/query?query=sum(rate(istio_requests_total{reporter="destination", source_workload="frontend", destination_workload="backend"}[5m]))'
This query looks at Prometheus, finds all istio_requests_total metrics where the reporter is the destination (meaning the backend sidecar recorded it), the source workload is frontend, and the destination workload is backend, and sums the rate over the last 5 minutes.
Distributed Tracing: Istio integrates with tracing backends like Jaeger or Zipkin. When mTLS is enabled, Istio automatically propagates tracing headers. This allows you to follow a single request as it traverses multiple microservices.
To enable tracing, you’d typically deploy Jaeger and configure Istio to send traces to it. This is often done via Istio’s meshconfig or by applying specific Istio operator configurations. Once set up, you’d see trace spans for each hop: ingress gateway -> frontend -> backend.
Access Logs: Istio sidecars generate access logs for every request. These logs, often forwarded to a centralized logging system like Elasticsearch/Fluentd/Kibana (EFK) or Loki, provide detailed information about each request, including source, destination, timestamps, HTTP status codes, and request duration.
You can configure access logging via meshConfig.accessLogFile in the Istio operator or istio-system ConfigMap.
The Counterintuitive Truth About mTLS Certificates
Most people assume that when mTLS is enabled, Istio is managing and rotating complex X.509 certificates for every single service. While it is managing certificates, the actual trust anchor for the entire mesh is a single Certificate Authority (CA) managed by Istio itself. The sidecars are issued certificates signed by this mesh CA. This dramatically simplifies certificate management compared to a scenario where each service had to manage its own CA or procure certificates from an external authority. The root of trust is centralized within the Istio control plane, making it robust and manageable at scale.
The next step is often dealing with complex authorization scenarios, like allowing specific HTTP methods or paths between services, or configuring rate limiting for services under heavy load.