Rate LimitingDistributed SystemsAPI DesignRedissenior level

Designing a distributed rate limiter

the question

Design a rate limiter for a public API used by millions of clients. How would it handle bursts, multiple service instances, and failures?

what they're testing: The interviewer is probing whether you can turn a traffic policy into a low-latency distributed design while reasoning clearly about correctness, scale, and failure modes.

a strong answer

Put the limiter before the application and make one fast allow-or-reject decision for every request. Start by defining the policy: who is limited, which routes share a quota, the sustained rate, and the allowed burst. The key is usually an authenticated user, API key, or tenant combined with a route or plan; IP can serve as a fallback.

For an API that tolerates short bursts, a token bucket is a good default. Tokens refill at a fixed rate up to a capacity, and each accepted request consumes one. In a distributed fleet, keep bucket state in a shared low-latency store such as Redis. The read, decision, and update must be atomic so concurrent requests cannot spend the same token. Per-process counters only enforce a per-instance limit and become inconsistent behind a load balancer.

When the bucket is empty, return 429 Too Many Requests with a useful explanation and, when possible, Retry-After. Layer per-client limits with route and global caps so one tenant cannot exhaust shared capacity. Track allowed and rejected requests, decision latency, and store errors. If the shared store fails, choose fail-open or fail-closed from the protected endpoint's risk, and make that behavior explicit.

Where people slip

the tempting wrong answer, and what's actually true

A counter in each application instance enforces one global quota.
Local counters enforce separate per-instance quotas, so distributed instances need shared or coordinated state for one global limit.
A token bucket prevents all traffic bursts.
A token bucket deliberately permits bursts up to its bucket capacity while enforcing a steady refill rate.
Calling Redis `INCR` and then `EXPIRE` separately is race-free.
Each command is atomic, but the two-command sequence is not. If the client stops after `INCR`, the key can remain without a TTL; use a transaction or Lua script that performs the update and expiration together.
Every `429 Too Many Requests` response must include `Retry-After`.
A `429` response may include `Retry-After`, but the HTTP specification does not require that header.

If they push further

How would you enforce the limit across multiple regions?

First clarify whether the quota must be globally strict. A central authority gives a tighter limit at the cost of cross-region latency and availability. Pre-allocated regional budgets keep decisions local without overshoot, but can strand unused quota; asynchronous reconciliation uses capacity better but can overshoot during lag or a partition.

What happens when one customer becomes a hot key?

Normal key-based sharding spreads different customers, but it cannot split one hot key. Isolate that tenant on dedicated capacity, or lease bounded sub-budgets to instances or regions to reduce contention. A strictly current global quota still requires coordination with one authority.

Should the limiter fail open or fail closed if Redis is unavailable?

Tie the choice to impact: fail open when availability matters more than temporary overuse, and fail closed when excess traffic creates serious security or cost exposure. A small local emergency allowance can preserve some availability, but it is multiplied across instances and is not a strict global limit.

Sources

Now answer it yourself.

Reading a strong answer is easy. Producing one under pressure is the skill the interview tests. Gapmap grades your answer against the same bar an interviewer would.

was this useful?

beta

The interviewer part is in the works.

The diagnostic, personal maps, and AI mock interviews are being finished right now. The notes stay free either way. Leave an email and you'll get the first-cohort invite, plus a month of Pro when it opens.