Service mesh

What a mesh adds and where it sits

A service mesh moves the cross-cutting parts of service-to-service networking (encryption, identity, retries, metrics) out of your application and into a proxy that runs beside it. In the classic sidecar model that proxy, usually Envoy, is injected as a second container in every pod, and the pod's traffic is rerouted through it by iptables or a CNI plugin. Your code still calls http://payments:8080 as before; the proxy quietly intercepts the call on the way out and the peer's proxy intercepts it on the way in.

Because the proxies own both ends of the wire hop, they can encrypt it, attach an identity to it, and count it, none of which your app has to know about. The control plane (istiod in Istio, the Linkerd control plane in Linkerd) configures those proxies and hands out their certificates. The proxies are the data plane; the control plane never sits in the request path.

Automatic mTLS: real, and narrower than it sounds

The headline feature is mutual TLS between services with no code change. The control plane runs an internal certificate authority, mints a short-lived certificate for each workload's identity, and rotates it automatically, so every proxy-to-proxy hop is encrypted and both sides know who the other is. The TLS handshake and certificate chain mechanics themselves live in the tls-and-certificates module; here the point is that the mesh automates the issuance and rotation that make per-workload mTLS practical at all.

One thing trips people constantly. mTLS proves identity, it does not grant permission.

Wrong "We turned on STRICT mTLS, so now only authorized services can reach payments."

Right mTLS answers "who is calling?" (an encrypted channel plus a verified identity). It does not answer "is this caller allowed to call payments?". Any workload in the mesh with a valid identity can still reach payments until you add an authorization policy that allows or denies by identity, namespace, or path. Authenticate with mTLS, authorize with a separate policy.

Istio exposes the enforcement as a mode: STRICT (accept only mTLS), PERMISSIVE (accept both mTLS and plaintext), or DISABLE. PERMISSIVE exists for migration, and you want it: flipping a whole cluster to STRICT at once cuts off every caller that is not injected yet.

Shifting traffic for a canary

The mesh can route by weight. In Istio a VirtualService splits traffic across subsets defined in a DestinationRule:

http:
- route:
  - destination: { host: reviews, subset: v1 }
    weight: 90
  - destination: { host: reviews, subset: v2 }
    weight: 10

That sends 10% of requests to the new version without scaling anything, which is the mechanism under a canary. Judging the canary while it runs (a concurrent control, the right metric, when to roll back) is release strategy, and it lives in feature-flags-and-progressive-delivery. The mesh gives you the traffic knob; the strategy decides how to turn it.

The same machinery does two other useful things. Mirroring (shadowing) sends a copy of live traffic to a new version while the real response still comes from the old one, so you can test with production load and no user impact. Fault injection adds delays or returns errors on a chosen fraction of requests, so you can check that callers time out and degrade the way you assumed.

The cost you sign up for

None of this is free. A sidecar mesh puts a proxy in every pod, so you pay its CPU and memory per replica, and you take on a control plane to run, upgrade, and debug. A misrouted VirtualService or a botched mTLS mode is a new class of outage that has nothing to do with your code. At a handful of services the honest answer is often that you do not need a mesh yet: NetworkPolicy for segmentation, TLS at the ingress, and the retry and metrics libraries you already have can cover the same ground with fewer moving parts. Reach for a mesh when the number of services and the need for uniform mTLS, traffic control, and telemetry across all of them outgrows doing it by hand.

Sidecar vs ambient: where the cost goes

The sidecar model's cost is structural: one Envoy per pod. At a thousand replicas that is a thousand proxies to schedule, and every pod restart to pick up a proxy upgrade is a rolling restart of the app. Ambient mode (GA in Istio 1.24, November 2024) splits the data plane to cut that. A per-node proxy called ztunnel, written in Rust and scoped to L3/L4, carries mTLS, L4 authorization, and telemetry for every pod on its node, so enabling the mesh is a namespace label rather than a pod injection. When you need L7 features (weighted routing, per-request authorization, HTTP-level telemetry) you add a waypoint proxy per namespace, an Envoy that scales like any other Deployment.

The secure overlay between ztunnels is an HTTP CONNECT tunnel called HBONE, so mTLS still terminates proxy to proxy. What changes is the economics: one shared waypoint can front a workload with many replicas instead of every replica carrying its own sidecar, and Istio reports ambient L4 latency well under sidecar's. The tax drops but you still run a control plane, and ambient's own status matters (single-cluster went stable first, multi-cluster trailed), so pin what your version actually supports rather than assuming parity with sidecars.

What mTLS does not protect

Automatic mTLS is genuinely strong for one thing: it replaces bearer tokens with a key-bound identity per workload, so a leaked token is useless without the private key the proxy holds. Seniors are expected to know its edges.

• It authenticates, it does not authorize. STRICT PeerAuthentication only requires that peers present mTLS. Restricting who may call what is an AuthorizationPolicy, a separate object that allows or denies by principal, namespace, path, or method. A mesh full of valid identities with no authorization policy is a flat trust domain.
• It covers the wire between proxies, not the whole path. The hop between your app and its own proxy is loopback inside the pod, outside the mTLS tunnel, and anything off the mesh (an un-injected pod, an external client, egress to a third party) is not covered at all. "The mesh encrypts everything end to end" is false in a way that matters for a threat model.
• A compromised but valid workload still passes. mTLS proves the caller is who its certificate says; it says nothing about whether that workload has been popped.
• For opaque TCP, secure naming cannot stop DNS spoofing, because the proxy has only the destination IP to route on. The identity guarantees that hold for HTTP do not automatically extend to arbitrary TCP.

Wrong "Once the mesh is on and STRICT, the internal network is zero-trust and we are done."

Right mTLS gives you the authenticated transport that zero trust needs, then authorization policies do the actual trust decisions. Turning on mTLS without writing those policies encrypts the traffic and authorizes nothing.

Retries, timeouts, and the retry storm

The mesh can retry failed requests at the proxy, and this is where a confident wrong answer does real damage. Istio's default is to retry twice; setting retries.attempts higher on every route feels like more reliability and is a way to build a self-inflicted outage.

The failure is amplification. Linkerd's design writeup spells it out: three retries per request can quadruple the traffic a struggling service sees, and when the callers of those callers also retry, the multiplication cascades into a denial of service you inflicted on yourself. A fixed per-request count has no awareness of how much of the fleet is already failing, so it hits hardest exactly when the system is weakest.

The better model is a retry budget: cap retries as a fraction of live traffic rather than per request. Linkerd's default allows retries to add at most about 20% to the request load, plus a small per-second floor (roughly 10) so low-traffic routes can still retry, over a rolling window. A cluster-wide failure then cannot balloon load past that fraction. Three rules travel with it:

• Retry only idempotent routes. The proxy re-sends the request and adds no deduplication, so a retried POST that charges a card charges twice. Gate non-idempotent routes out explicitly.
• Bound each attempt with a per-try timeout, so a retry of a request that is merely slow does not stack latency on top of latency.
• Compose retries with outlier detection. Envoy ejects a host after a configured run of errors (consecutive_5xx and friends), grows the ejection time on repeat offenders, and fails open once too many hosts are ejected. The ejection theory is the load-balancing-and-proxies module's ground; the mesh angle is that outlier detection is what keeps retries from re-landing on the host that is already down, so the two are configured together per DestinationRule.

Observability: metrics free, tracing needs a header

The proxies see every request, so a mesh gives you golden-signal metrics (rate, errors, duration) per service and access logs with no code change, which is the genuinely free part. Distributed tracing is where the "zero changes" claim breaks. The proxy can emit a span for the hop it handles, but it cannot stitch spans into one trace unless the application forwards the trace context (B3, or W3C traceparent) from the inbound request to its outbound calls. Drop the header in your code and every service produces disconnected single-hop spans instead of one end-to-end trace. Tracing is the mesh doing most of the work and still needing a small, real change in every service.

The other limit is protocol. L7 metrics and tracing depend on the proxy parsing the protocol; treat a connection as opaque TCP and you get connection-level counters, not per-request rates, errors, and latency. A service speaking a protocol the proxy does not understand drops back to L4 visibility.

When not to mesh

The operational tax is the deciding factor more often than the feature list. A mesh is a control plane to run and upgrade, a proxy in the request path of every service, and a new set of failure modes (a bad VirtualService, an mTLS mode mismatch, a proxy that outpaced a control-plane upgrade) that are unrelated to your application logic. Below some service count, and small teams without a platform or SRE function feel this first, the mesh is more system to operate than problem solved. NetworkPolicy handles segmentation, ingress handles edge TLS, and the retry and metrics libraries already in the app cover the rest, without a mesh in the path.

When a mesh is justified, adopt it the way you would flip mTLS: incrementally. One namespace, PERMISSIVE before STRICT, a few services before all of them, with authorization policies written as you go rather than bolted on after. Ambient lowers the per-pod cost enough to change the threshold, but the question it does not remove is whether uniform mTLS, traffic control, and telemetry across your services is worth running another distributed system to get them.