The most surprising thing about Istio’s header manipulation is that it’s fundamentally a routing operation, not a transformation. You’re not changing the header in place; you’re telling Istio to send a new request downstream with different headers, while the original request from the client is effectively discarded.
Let’s see this in action. Imagine you have a simple service my-service running on port 8080, and you want to add a custom header X-My-Header with the value my-value to all incoming requests before they hit my-service.
Here’s the VirtualService that does it:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-service-header-manipulation
spec:
hosts:
- my-service.default.svc.cluster.local # The service Istio will route to
gateways:
- mesh # Apply this rule to traffic within the mesh
http:
- route:
- destination:
host: my-service.default.svc.cluster.local
port:
number: 8080
headers:
request:
add:
- name: "X-My-Header"
value: "my-value"
When a request arrives at Istio’s ingress gateway (or sidecar) destined for my-service, Istio intercepts it. Instead of forwarding the original request directly to my-service, it constructs a new request. This new request is identical to the original in terms of its method, path, and query parameters, but it includes the X-My-Header: my-value header that you specified. The original request, with its original headers, is never seen by my-service.
This routing perspective is crucial for understanding how to build a robust mental model of header manipulation. You’re not modifying headers in transit; you’re defining a rewrite rule that produces a new request.
Let’s break down the VirtualService YAML:
hosts: This specifies the internal Kubernetes service name that thisVirtualServiceapplies to. Traffic for this host will be subject to the rules defined here.gateways: In this case,meshmeans this rule applies to all traffic flowing within the Istio service mesh. You could also specify specific ingress gateways.http: This section contains the routing rules for HTTP traffic.route: This defines where the traffic should be sent after any header manipulations. Here, it’s pointing tomy-serviceon port 8080.headers: This is where the magic happens.request: Indicates that we are manipulating headers on the request side (client to service). You can also doresponseheaders.add: A list of headers to add. Each entry has anameand avalue.remove: A list of header names to remove from the incoming request before forwarding.rewrite: This is more powerful. You can specify asourceheader and adestinationheader. The value of thesourceheader is used to set the value of thedestinationheader.
Let’s say you want to remove the X-Forwarded-Proto header and then rewrite the Host header to internal.example.com:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-service-complex-headers
spec:
hosts:
- my-service.default.svc.cluster.local
gateways:
- mesh
http:
- route:
- destination:
host: my-service.default.svc.cluster.local
port:
number: 8080
headers:
request:
remove: ["X-Forwarded-Proto"]
rewrite:
source_labels: # You can also use source labels to derive the value
app: my-app
headers: # This is for rewriting the Host header
- name: "Host"
value: "internal.example.com"
This VirtualService will first remove X-Forwarded-Proto. Then, it will construct a new request where the Host header is explicitly set to internal.example.com. The original Host header from the client is discarded, and the Host header value used by my-service will be internal.example.com.
The rewrite section is where you see the true power of Istio. You can dynamically set header values based on request properties. For example, you can set a header based on the value of another header:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-service-dynamic-rewrite
spec:
hosts:
- my-service.default.svc.cluster.local
gateways:
- mesh
http:
- route:
- destination:
host: my-service.default.svc.cluster.local
port:
number: 8080
headers:
request:
rewrite:
headers:
- name: "X-Original-User-Agent"
value: "User-Agent" # Copies the value of the incoming User-Agent header
- name: "X-Tenant-ID"
value: "tenant-abc" # Sets a static value
In this example, the X-Original-User-Agent header will be populated with whatever was in the incoming User-Agent header. The X-Tenant-ID will always be tenant-abc. This is still a rewrite operation: Istio is constructing a new request with these headers, not modifying the original one in place.
What often trips people up is the order of operations when add, remove, and rewrite are all used together. Istio processes these in a specific, but predictable, sequence. It first removes headers, then rewrites them (using source labels or other headers), and finally adds any new headers. This ensures that if you remove a header and then try to rewrite it, the rewrite will fail because the source header no longer exists.
If you want to add a header that might already exist, but you want to ensure it has a specific value, you should use rewrite to set the desired value and then add it if it’s not already present. However, the simplest way to ensure a header has a specific value is to rewrite it. If it exists, its value is replaced. If it doesn’t exist, it’s added with that value. The add directive is purely for adding new headers that are guaranteed not to exist, or if you want to append to a multi-valued header (though Istio’s add directive doesn’t directly support appending to multi-valued headers in the way some protocols do; it will simply add another instance of the header).
The next thing you’ll likely encounter is wanting to conditionally add or modify headers based on the value of an incoming header, not just its presence.