Load balancing and proxies

L4 vs L7: what each can see

A load balancer sits in front of a pool of backends and spreads traffic across them. Where it sits in the stack decides what it can read.

A Layer 4 balancer works at the connection level: source and destination IP and port. It picks a backend and forwards the packets without looking inside them. It never sees a URL, a header, or a cookie, because those live in the HTTP request it is not parsing. Cheap and fast, but blind to content.

A Layer 7 balancer terminates the client connection, reads the full HTTP request, then opens its own connection to a backend. Because it parses the request, it can route on the path, the Host header, a cookie, or any header, and it can retry a failed request on another backend. That parsing costs CPU on every request.

So the rule that answers most interview questions: if the routing decision depends on what the request says (send /api here and /images there), you need L7. If you are just spreading raw connections as cheaply as possible, L4 is enough. AWS makes the split concrete: the Application Load Balancer is L7 (HTTP/HTTPS), the Network Load Balancer is L4 (TCP/UDP).

Balancing algorithms and when to use each

The algorithm decides which backend in the pool gets the next request.

• Round-robin: rotate through the backends in order. The default in nginx and on an AWS ALB. It assumes every request costs about the same and every backend is about equally powerful.
• Least-connections (nginx least_conn, AWS calls it "least outstanding requests"): send the next request to the backend with the fewest in-flight requests. Better when request durations vary a lot, because one slow request holds its slot and steers new traffic elsewhere.
• Weighted: give a bigger box a higher weight so it receives proportionally more traffic. Round-robin and least-connections both respect weights.
• Hash-based: hash a key (client IP, a URL, a user id) so the same key always lands on the same backend. Used for affinity, for example keeping a cache key on the node that already has it warm. The stable version is consistent hashing; the ring mechanics and why plain hash % N is bad are their own topic (see consistent-hashing).

Pick least-connections when some requests are much slower than others; pick a hash when the same key should stick to the same backend; round-robin is the sane default otherwise.

How health checks pull a bad backend

The balancer only sends traffic to backends it believes are healthy. There are two ways it forms that belief.

A passive check watches real traffic: if requests to a backend start failing, mark it down for a while, then try it again. nginx Open Source does this with two server parameters:

upstream app {
    server 10.0.0.1 max_fails=3 fail_timeout=30s;  # 3 failures in 30s -> down for 30s
    server 10.0.0.2 max_fails=3 fail_timeout=30s;
}

An active check probes each backend on a schedule (hit /health every few seconds, mark it down after N failures) whether or not real traffic is flowing. The catch that trips people: active checks in nginx are an NGINX Plus feature. HAProxy and Envoy do active checks in their free builds; nginx Open Source gives you passive only.

Wrong "My nginx has max_fails set, so it is probing the backends and will catch a sick one before users do." That is passive. nginx marks a backend down only after live requests to it fail, so a backend that is broken but idle stays "healthy" until a real user hits it. Right to probe proactively on nginx you need Plus (health_check), or you use HAProxy or Envoy, which probe in open source.

Reverse proxy vs forward proxy

Same machinery, opposite direction, and interviewers do ask which is which.

A reverse proxy sits in front of your servers and faces the clients. Clients think they are talking to one host; the proxy fans their requests out to the backends behind it and returns the responses. A load balancer is a reverse proxy. In nginx it is one directive:

location /api/ {
    proxy_pass http://app_backend;  # nginx receives the request, forwards it to the pool
}

A forward proxy sits in front of the clients and faces the internet. The client sends its outbound requests through the proxy, which reaches the origin on its behalf (a corporate egress proxy, a filtering gateway). A reverse proxy hides the servers from the client; a forward proxy hides the clients from the server.

Sticky sessions and why to avoid them

Sticky sessions (session affinity) pin a user to one backend, usually with a cookie the balancer sets. The first request picks a backend; every later request from that user goes back to the same one. On an AWS ALB the balancer-generated cookie is AWSALB.

People reach for stickiness to keep a login session that a node stored in its own memory. It works until it does not:

• when that node dies, every session it held dies with it, and those users are logged out;
• load skews toward whichever nodes happen to hold the chatty users;
• deploys get harder, because you cannot restart a node without dropping its sessions.

The durable fix is to stop keeping the session in one node's memory: write it to a shared store (Redis, a database, or a signed token the client carries) so any node can serve any user. Then you do not need stickiness at all. Sometimes you keep affinity on purpose (to reuse a warm in-memory cache), but with the shared store as the backstop so a reconnect recovers. Treat "we need sticky sessions" as a signal that state is living somewhere it should not.

What L7 buys, and what it costs

An L7 balancer is a full HTTP intermediary: it terminates the client's TCP and TLS, parses the request, applies rules, then originates a fresh connection to a backend and proxies the response back. Two independent connections, one on each side. That is the source of everything L7 can do that L4 cannot: route on path or Host or header, rewrite headers, retry an idempotent request on another backend when the first fails, buffer a slow client so the backend is not held open, and terminate TLS centrally. It is also the cost: every request is parsed and re-emitted, and the balancer, not the client, is now the TCP peer the backend sees, so the backend loses the real client IP unless you carry it (X-Forwarded-For, or PROXY protocol on L4).

L4 gives that up for throughput. It forwards at the connection level, can pass traffic it cannot read (any TCP or UDP protocol), sustains far higher packet rates, and can preserve the source IP by design. A common shape at scale is L4 at the edge spreading connections cheaply into L7 pools that do the content-aware work.

Wrong "An L7 balancer is just a faster L4 with more features." It is slower per request precisely because it reads and rebuilds every request; you pay CPU for the visibility. Reach for L4 when you need raw connection throughput or a non-HTTP protocol, L7 when the decision depends on request content.

Least request and power-of-two-choices

Round-robin is correct only when requests cost about the same. When they do not, least-request routes to the backend with the fewest in-flight requests, which self-corrects around slow backends. The naive version scans every backend for the true minimum on each pick. Envoy's default instead uses power-of-two-choices: sample two backends at random and send to the less loaded of the two. That is O(1) per decision and, unlike a global "pick the absolute least loaded", it resists the herd effect where every balancer instance stampedes the one backend that briefly looked idle. Envoy's weighted variant folds weight in as weight / (active_requests + 1)^bias.

Hash-based methods (nginx hash ... consistent, Envoy ring_hash with ketama, or maglev) exist for affinity, not spreading: the same key lands on the same backend so a cache stays warm or a shard stays put. The ring math and why hash % N remaps almost everything on a resize belong to the consistent-hashing topic; here the point is only that you choose a hash when you need key affinity and accept that it does not balance load evenly on its own.

Why long-lived connections defeat an L4 balancer

This is the failure that bites HTTP/2, gRPC, and WebSocket fleets. A single HTTP/2 connection multiplexes many concurrent requests as streams over one TCP connection, and a WebSocket is one connection held open for the session. An L4 balancer places the connection once, at open time, and then every request on it rides to the same backend forever. Scale the backend pool out and the new nodes sit idle, because no new connections are being made: the existing ones just keep carrying traffic to where they first landed.

Least-connections does not save you here, and this is the subtle part interviewers probe. At L4 the counter sees one connection per client; it cannot see the hundreds of multiplexed streams inside it. A client hammering the service with 500 concurrent streams and an idle client both register as "one connection". The fix is to balance where the requests actually are, or to stop the connections from being immortal:

• put an L7 proxy in the path that picks a backend per request (an Envoy sidecar, or an L7 balancer that speaks HTTP/2);
• bound connection lifetime so clients re-spread: cap max concurrent streams so the client is forced to open more connections, set idle timeouts, or have clients periodically reconnect.

gRPC-specific balancing (client-side subchannels, lookaside) is its own topic; the connection-pinning mechanism above is what all of those approaches are working around.

Active checks, outlier detection, and failing open

Two independent mechanisms decide which backends are in rotation.

Active health checks probe on a schedule, independent of traffic. HAProxy: add check to a server line, tune inter (interval, default 2s), rise (successful checks to come back up, default 2), and fall (failed checks to go down, default 3); option httpchk sends an HTTP probe. Envoy runs active checks natively. nginx does active checks only in Plus (health_check). The value is that a backend which is sick but receiving little traffic still gets tested.

Passive checks (Envoy calls the richer version outlier detection) judge backends from real traffic. Envoy ejects a host after enough consecutive errors (consecutive_5xx) and returns it after an ejection timeout, and the ejection window grows each time a host is re-ejected. A successful active health check un-ejects a host and clears its outlier counters, so the two mechanisms compose. Run both: active to catch the idle-but-broken node, passive/outlier to catch the node that passes a shallow probe but errors on real requests.

Wrong "The balancer will always route around unhealthy nodes, so more failures just means more routing-around." Every serious balancer fails open past a point. Envoy's healthy panic threshold defaults to 50%: once fewer than half the hosts look healthy it disregards health and routes to all of them, because sending all traffic to the two survivors would just collapse them too. Ejection is bounded on purpose; do not assume it is unlimited. On the way back, slow-start ramping (HAProxy slowstart, nginx Plus slow_start) or ALB slow-start ramps a recovered node's share from zero so it is not flooded the instant it returns.

Draining connections for a graceful deploy

The 502-storm-on-deploy is almost always a missing drain. If you terminate a node while it still holds in-flight requests and long-lived connections, those connections are severed and clients see resets and 502s. A graceful roll is an ordered sequence, and the order is what candidates miss:

First flip the node out of the healthy set (fail its health check, or deregister it) so the balancer stops opening new connections to it. The node enters a draining state: it keeps serving the requests and connections it already has but accepts no new ones. You wait out a drain window, then stop the process. AWS calls this deregistration delay and defaults it to 300 seconds; during that time the target sits in draining, and if it kills its connections before the window elapses the client gets a 500-level error, which is the whole thing you are trying to avoid. The application has to cooperate: catch SIGTERM, stop accepting new work, finish or cleanly close what is open, then exit. For long-lived connections a hard drain timeout eventually wins, so pair draining with a client that reconnects gracefully.

nginx, HAProxy, and Envoy

The three you will be asked to compare. nginx started as a web server and reverse proxy and is the default for terminating TLS and serving or proxying HTTP; its load-balancing and active health checks are strongest in the paid Plus tier. HAProxy is a dedicated L4/L7 balancer with deep health checking, draining, and observability in the open-source build, long the workhorse for high-traffic edges. Envoy is the newer L7 proxy built for service meshes: dynamic configuration over its xDS APIs (endpoints and routes pushed at runtime, no reloads), per-request HTTP/2 and gRPC balancing, outlier detection, and it is the data plane under Istio. The rough mapping: nginx for a web-server-plus-proxy, HAProxy for a heavy standalone balancer, Envoy where you need dynamic, per-request, mesh-native routing.