Distributed systems

What CAP actually says, and what PACELC adds

CAP is narrower than the mnemonic suggests. Partition tolerance is not a dial you turn down: networks drop packets, so a real distributed system is always P. The theorem only bites during a partition, and then it forces one choice. Refuse requests you cannot make consistent (CP), or answer anyway with possibly stale data (AP). No partition, you get both.

That is why PACELC exists. Else (no partition) the live tradeoff is latency versus consistency: every synchronous replica you wait on before acknowledging a write costs milliseconds. Cassandra is PA/EL, a Spanner-style store is PC/EC.

One precision: CAP's availability means every request to a non-failing node returns a non-error response, not "four nines of uptime". A CP database is not less reliable in the marketing sense; it declines to answer while partitioned.

Which "consistency" word do you mean

Three unrelated meanings share the word, and the confusion is graded.

• Linearizability: single-object recency. A read sees the most recently completed write.
• Serializability: multi-object transaction isolation. Some serial order of the transactions exists.
• ACID's C: no declared constraint is violated, a database-integrity property with nothing to do with CAP.

Wrong "we run serializable isolation, so our reads are linearizable." Serializable can still hand you a stale snapshot; strict serializability is both. Most product bugs only need a session guarantee: read-your-writes (you always see your own writes), monotonic reads (time never runs backward for one client). Eventual consistency promises only that replicas converge once writes stop, with no bound on when.

Consensus, quorums, and why clocks lie

Consensus (Raft, Paxos) gets a set of nodes to agree on one ordered log despite crashes. It needs a majority quorum so any two quorums intersect; lose the majority and it goes unavailable rather than inconsistent, the CP choice spelled out. You do not implement it in an interview; you know it exists, what it guarantees, and that it costs a round trip to a majority.

Ordering is the other half, and wall clocks are the trap. They drift and can jump backward, so timestamps cannot order events across machines. Last-write-wins by wall clock silently drops the write whose node had a slow clock. Logical clocks fix this: Lamport timestamps give a total order (a before b implies L(a) < L(b), not the reverse), and vector clocks detect that two writes were concurrent, so you merge instead of discard.

This is also why at-least-once delivery plus idempotency beats chasing exactly-once: the acknowledgement can be lost after the work is done (Two Generals), so you make the operation safe to repeat. Each of these traps is a line from the eight fallacies of distributed computing: the network is reliable, latency is zero, the network is secure, topology is stable, and the rest. When an outage post-mortem lands, the fastest diagnosis is naming which fallacy the code assumed.