Istio’s traffic mirroring sends a copy of live production traffic to a separate destination without impacting the original request, acting like a read-only snapshot for testing.
Let’s see it in action. Imagine you have a reviews service, and you want to shadow its production traffic to a new reviews-v2 service for testing.
First, you need a VirtualService that routes traffic to your reviews service.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews-vs
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v1
weight: 100
Now, to mirror traffic, we add a mirror directive to this VirtualService. This tells Istio to send a copy of all traffic matching this route to the reviews-v2 service.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews-vs
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v1
weight: 100
mirror:
host: reviews-v2
subset: v2
In this configuration, reviews-v2 will receive a duplicate of every request that goes to reviews:v1. The response from reviews-v2 is discarded and never sent back to the client. The client only sees the response from reviews:v1. This is critical: mirroring is non-intrusive to the end-user.
This setup solves a common problem: how to test new versions of a service with real-world traffic without risking user impact. Traditional methods like canary deployments gradually shift traffic, but mirroring allows you to observe behavior with 100% of live traffic without any risk.
The mirror directive can also be applied to specific http match conditions within a VirtualService. This is useful if you only want to mirror traffic for certain request patterns.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews-vs
spec:
hosts:
- reviews
http:
- match:
- headers:
user-agent:
regex: ".*testing.*"
route:
- destination:
host: reviews
subset: v1
weight: 100
mirror:
host: reviews-v2
subset: v2
- route: # This route handles all other traffic
- destination:
host: reviews
subset: v1
weight: 100
Here, only requests with a user-agent containing "testing" will be mirrored to reviews-v2. All other requests to reviews will proceed normally without mirroring.
You can also specify a mirrorPercentage to mirror only a fraction of the traffic. This is useful when you want to start with a small sample of mirrored traffic and gradually increase it.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews-vs
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v1
weight: 100
mirror:
host: reviews-v2
subset: v2
mirrorPercentage:
value: 25.0 # Mirror 25% of the traffic
When mirrorPercentage is set, Istio randomly selects requests to mirror. This percentage is applied to the total traffic destined for that specific route.
The destination for mirroring doesn’t have to be a different service version. You could mirror to a debugging service, a logging aggregator, or even another instance of the same service for redundancy checks. The key is that the mirrored traffic is a verbatim copy, including headers and body, allowing for deep inspection of how your service behaves under real load.
When you mirror traffic to a new service version, like reviews-v2, you typically deploy reviews-v2 with no traffic routed to it initially. You then use traffic mirroring to send it a copy of production traffic. You can then observe the logs, metrics, and error rates of reviews-v2 without any risk to your users. If reviews-v2 behaves as expected, you can then shift production traffic to it using standard Istio routing rules.
A common pitfall is forgetting to define the mirrored service (e.g., reviews-v2) in Istio’s DestinationRule. Without a DestinationRule, Istio won’t know how to route traffic to the mirrored destination.
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: reviews-v2-dr
spec:
host: reviews-v2
subsets:
- name: v2
labels:
version: v2
This DestinationRule tells Istio that reviews-v2 has a subset named v2 and how to select pods for it (based on the version: v2 label).
The actual processing of mirrored traffic happens within the Envoy sidecar proxy. When a request arrives at the source service’s sidecar, Envoy inspects the VirtualService configuration. If mirroring is enabled for that route, Envoy makes a separate, asynchronous call to the mirror destination. The original request continues to its intended destination, and its response is returned to the client. The response from the mirrored destination is ignored. This asynchronous nature is what prevents mirroring from impacting the latency or success rate of the primary request.
If you’re mirroring traffic to a service that’s not ready or has issues, it won’t affect the primary service’s availability. This isolation is the core benefit of traffic mirroring for testing.
After you’ve validated your mirrored service and are ready to route production traffic to it, you’ll typically adjust the weight in the VirtualService to shift traffic gradually or entirely.