Scaling & load balancing

Horizontal vs vertical scaling

Vertical scaling means giving one machine more CPU, RAM, or disk. It is the fast fix (no code changes) but it caps out at the largest instance you can buy, and that box is still a single point of failure. Horizontal scaling means running many identical machines behind a load balancer and splitting traffic across them. No hard ceiling, redundancy for free (one node dies, the balancer routes around it), and you can deploy by rotating nodes in and out. The catch: it only works if any node can serve any request, and that one requirement is where the real work lives.

Why services must be stateless to scale out

A stateless service keeps nothing about a user between requests in its own memory. If server A stores your login session in a local variable and the load balancer sends your next request to server B, server B has never heard of you and bounces you to the login screen.

const sessions = new Map(); // per process — invisible to other nodes
app.post("/login", (req, res) => { sessions.set(token, user); /* on A only */ });
app.get("/me", (req, res) => sessions.get(req.cookies.sid)); // 401 on B

Wrong "The load balancer remembers which user is which and keeps them consistent." A round-robin balancer just rotates through nodes. It does not read your session or share memory between servers. The fix is to move state out of the process: put sessions in a shared store like Redis so every node reconstructs your session from the cookie. The state did not disappear, it moved to a tier that all nodes can reach.

Sticky sessions vs a shared session store

Two ways to handle sessions once you have many nodes:

• Sticky sessions: the balancer pins a user to the same server (usually by a cookie). That server keeps their session in memory. Simple, but fragile: when that node dies its sessions die with it, load skews toward whichever nodes hold the chatty users, and rolling deploys have to drain sessions before restarting.
• Shared store: sessions live in Redis or Memcached, and every app node is stateless. Any node serves any user. This is the default for anything that needs to scale. The cost is a network hop per session read and a store you now have to keep highly available.

Read replicas for read-heavy load

When reads dominate, point writes at the primary and fan reads out across read replicas (copies of the DB kept in sync from the primary). Replication is usually asynchronous: the primary confirms your write before the replicas have applied it. So a read right after a write can hit a replica a few hundred milliseconds behind and return the old value. That is replication lag, and it surprises people who expect a read to always see their own write. If a user needs to see their own change immediately, route their reads to the primary for a short window. Replicas scale reads, not writes; scaling writes is a different problem (sharding).

L4 vs L7 load balancing

A Layer 4 balancer routes on IP and port without parsing the payload. It picks a backend and forwards packets at the connection level, so it burns little CPU, sustains high packet rates, and adds predictable latency. A Layer 7 balancer terminates the client connection, reads the HTTP request, and opens its own connection to a backend. That unlocks routing by path, host, header, or cookie, plus TLS termination and per-request retries, all of which cost CPU because every request is parsed. The common shape at scale is L4 in front spreading connections cheaply, with L7 pools behind it doing content-aware routing. Reach for L7 when the routing decision depends on what the request says.

Algorithm choice follows the workload. Round-robin assumes requests cost about the same. Least-connections wins when durations vary widely, since one slow request holds its connection and steers new traffic elsewhere. Consistent hashing is for affinity: the same key (user, cache entry, shard) should land on the same backend.

Consistent hashing, and why it matters here

Naive hash(key) % N remaps almost every key when N changes, so adding one cache node cold-misses the whole tier. Consistent hashing places nodes and keys on one hash ring; a key belongs to the next node clockwise. Add or remove a node and only the keys in that one arc move, roughly 1/N of them. Give each physical node many virtual nodes on the ring so load spreads evenly instead of clumping wherever a node landed. That is why a cache client fronting a sharded cache uses consistent hashing: membership changes stay cheap.

The limits of statelessness

Stateless does not mean the state vanished. It means the process holds no session between requests; the state moved to a shared tier (session store, database, object storage, a queue). You relocated the problem, and that tier is now the thing that has to be highly available and fast.

Wrong "Make it stateless and you have removed the state problem." You have moved it. Sometimes you keep affinity on purpose: sticky routing to preserve an in-memory cache's warmth or a live WebSocket, with a shared store as the durable backstop so a node loss degrades rather than drops. The senior move is naming where the state went and what now guarantees it.

Read replicas: what breaks and how graders probe it

Replicas scale reads, not writes. Async replication is the default: the primary commits and acks before replicas apply the change, so replicas serve slightly stale data. That produces the read-your-writes anomaly, where a user edits their profile and their next read hits a lagging replica showing the old value. Fixes, cheapest first: route a writer's reads to the primary for a window after their write, pin a session to one replica so it never sees time go backward, or make replication synchronous (Postgres synchronous_commit, Redis WAIT) at the cost of write latency if a replica stalls. Write scaling is a separate problem (sharding); do not conflate it with replicas.

Interviewers grade a few concrete moves: trace how a request finds a backend, choose an algorithm for a stated workload and defend it, and volunteer replication lag and where session state lives before being asked. Health checks and connection draining come up too, because "just add servers" is really "add servers, then solve the coordination, discovery, and consistency problems you created."