Resilience patterns

Why every network call needs a timeout

A remote call has four outcomes, not two: it succeeds, it fails fast, it times out, or it succeeds but the reply is lost. The default HTTP client in most languages sets no timeout, so a slow dependency does not return an error. It just hangs, holding a thread and a connection. Once enough calls hang, your own pool is exhausted and your service stops answering requests it could have served. A slow dependency becomes your outage.

Set a timeout on every outbound call. There are two: the connection timeout (TCP and TLS handshake) and the request timeout (waiting for the response). Tune the request timeout to the dependency's real p99, not a round number. Too tight and you manufacture failures on healthy-but-slow calls; too loose and hung calls stack up.

Retries with backoff and jitter

Transient failures are common: a dropped packet, a brief 503, a node restarting. A retry papers over those. The naive version does not:

// WRONG: retry immediately, forever, on any error
while (true) {
  try { return await call(); }
  catch { /* loop again right now */ }
}

That hammers a struggling service at full speed. Exponential backoff spaces the attempts out: wait base, then 2·base, 4·base, capping both the delay and the attempt count. But backoff alone has a trap. If a hundred clients failed at the same instant, they all wait 100ms, all retry together, all wait 200ms together. The spike re-synchronizes. That is the thundering herd.

Jitter fixes it by randomizing each client's delay so the retries spread out:

const cap = 2000, base = 100;
const delay = Math.random() * Math.min(cap, base * 2 ** attempt); // Full Jitter

wrong "we use exponential backoff, so we're resilient." Without jitter the herd still retries in lockstep. Jitter is what actually decorrelates the fleet.

Retries need idempotency

A timeout does not tell you whether the work happened. The request may have reached the server, applied its effect, and had the success reply lost on the way back. Retry that and you apply the effect twice.

For a read or an idempotent write (a PUT that sets a value, a DELETE) the retry is harmless. For a non-idempotent write (a POST that charges a card) it can double-charge. Only retry operations that are idempotent or that carry an idempotency key the server dedupes on (api-design covers how that key is built). And only retry retryable failures: timeouts, connection errors, 503, 429. A 400 or 404 fails identically every time, so retrying it just wastes calls.

The circuit breaker

When a dependency is genuinely down, retrying every call wastes resources on requests that will fail. A circuit breaker tracks failures and short-circuits through three states. Closed: calls flow, failures are counted. Open: once failures cross a threshold, the breaker trips and calls fail fast for a cooldown without touching the dependency. Half-open: after the cooldown it lets a few trial calls through, closing on success and re-opening on any failure. Failing fast frees the threads that would otherwise block on a dead dependency, which is what stops one sick service from dragging down its callers.

How retries amplify an outage

The failure mode that separates seniors from the rest is retry amplification. Retries look local, but they compose. In a chain A calls B calls C, if every layer retries 3 times, C sees up to 3x3 = 9x its normal load, and it sees that surge precisely when it is already unhealthy. Retries multiply down the stack, so a small blip at the leaf becomes a stampede that keeps the leaf down.

Two fixes, usually together. Retry at ONE layer, typically the tier closest to the user, and let inner layers fail fast. And run a retry budget: cap retries to a small fraction of live traffic (Google SRE uses ~10%) rather than a per-call count, so a broad outage cannot triple your request volume. A circuit breaker upstream helps for a second reason beyond politeness to the dependency: an open breaker stops feeding retries into C at all.

wrong "exponential backoff prevents the amplification." Backoff spreads retries in time; it does not reduce how many there are. Nine calls arriving over two seconds instead of one still crush a service that can serve one.

Deadlines and bulkheads

A per-hop static timeout lets a request blow past the user's deadline: if the client gives up at 2s but each of five hops waits up to 2s, work continues on a request nobody is waiting for. Propagate a deadline instead. Each hop passes the REMAINING budget downstream and refuses work that cannot finish in time. gRPC does this natively.

Bulkheads bound the blast radius. Give each dependency its own thread pool, connection pool, or concurrency limit, so one saturated dependency cannot consume every thread and starve calls to healthy ones. Flood one compartment, the ship still floats.

# Resilience4j: trip on error RATE and on slow calls, not N in a row
circuitbreaker:
  failureRateThreshold: 50          # open if >50% of the window fails
  slowCallRateThreshold: 50         # a call slower than the threshold counts as a failure
  slowCallDurationThreshold: 2s
  minimumNumberOfCalls: 20          # need a sample before judging
  permittedNumberOfCallsInHalfOpenState: 5

Production breakers trip on error rate over a rolling window and count slow calls as failures, because a dependency that is slow but not erroring is exactly what exhausts your threads.

Backpressure, load shedding, and what gets graded

Under overload you have two distinct levers. Backpressure signals upstream to slow down: bounded queues, flow control, a 429 or 503 with Retry-After. Load shedding drops the least valuable work early and cheaply once you are already over capacity, so the requests you do accept still complete. The metric is goodput, useful work finished, not throughput. A server that accepts everything and finishes nothing has high throughput and zero goodput. Shed low-priority traffic first, keep a fallback for the rest (cached data, a default, an empty list) so a dependency failure degrades the response instead of failing the whole request.

Interviewers grade a few specific moves: setting a timeout on every call by reflex, pairing retries with idempotency and jitter unprompted, naming the circuit breaker states and why half-open probes rather than flips, and catching retry amplification in a call chain. Reaching for a retry budget or deadline propagation is the signal that you have run this in production, not just read the pattern.