Queues & async processing

Queue vs pub/sub vs log

A work queue splits messages across competing consumers: put 100 jobs in, run five workers, each job is handled once by one of them. That is how you parallelize slow work (emails, thumbnails, exports). Pub/sub is different: every subscriber gets its own copy of every message, so three services can each react to the same "order.placed" event.

RabbitMQ and Kafka sit on opposite sides of this. RabbitMQ is a broker that pushes messages to consumers and deletes each one once it is acked. Kafka is an append-only log: it keeps messages for a retention window (say 7 days) whether or not anyone read them, and each consumer tracks its own offset (a bookmark into the log). Because the data stays put, a new consumer can replay from the start and a bug fix can reprocess yesterday's events. An acked RabbitMQ message is gone.

At-most-once vs at-least-once delivery

You do not get to pick "exactly once" for free. The choice is where the acknowledgement sits relative to your work.

• Ack before processing: if the worker crashes mid-job, the message is already gone. You lose it. This is at-most-once.
• Ack after processing: if the worker crashes after doing the work but before acking, the broker redelivers. You process it twice. This is at-least-once.

Almost every real system runs at-least-once, because losing a payment is worse than charging twice, and the second problem you can actually solve.

wrong "we use at-least-once, so duplicates can't happen." At-least-once guarantees duplicates are possible. It only promises the message is not lost. Handling the duplicate is your job, not the broker's.

How to write an idempotent consumer

Idempotent means running the handler twice leaves the same end state as running it once. Two common ways:

-- dedup on a business/message id, in the same transaction as the effect
BEGIN;
INSERT INTO processed_events (event_id) VALUES ($1); -- unique constraint
UPDATE orders SET status = 'paid' WHERE id = $2;
COMMIT; -- a redelivered event_id violates the constraint and rolls back

If the event id is already there, the INSERT fails, the transaction rolls back, and the second attempt is a no-op. The dedup record and the side effect commit together, so a crash cannot leave one without the other.

Dead-letter queues and ordering

Some message always throws (bad data, a downstream that is down). Retrying it forever blocks the queue behind it. After a few attempts, route it to a dead-letter queue: a side queue you alert on, inspect, and replay after a fix. That quarantines the poison message so the rest keeps flowing.

Ordering only holds within a single queue or Kafka partition. The moment two consumers pull the same stream in parallel, they finish in whatever order they finish. If order matters, route one account's messages to a single partition so one consumer sees them in sequence.

Why exactly-once delivery is impossible

Two machines over an unreliable network cannot agree that a message was processed exactly once. The worker does the job and sends an ack; the ack is lost; the sender cannot tell "processed, ack dropped" from "never processed." It either resends (risking a duplicate) or gives up (risking a loss). That is the two-generals problem, and no protocol escapes it. So what ships is at-least-once delivery plus idempotent processing, which people market as "exactly-once" and engineers call effectively-once.

wrong "we turned on exactly-once, so the consumer can't double-charge." Kafka's exactly-once semantics are scoped to read-process-write pipelines that stay inside Kafka. The moment your effect leaves the cluster (charge a card, send an email, call another API) the guarantee ends and you are back to needing an idempotent consumer.

How Kafka exactly-once actually works

Two mechanisms, worth naming precisely in an interview. The idempotent producer tags every record with a producer id and a per-partition sequence number; on a retry the broker sees a sequence number it already stored and drops the duplicate write. Transactions then make a set of writes across partitions, plus the consumer offset commit, atomic: a reader with isolation.level=read_committed never sees records from an aborted transaction. Since Kafka 3.0 acks=all and enable.idempotence=true are the defaults. This gives genuine exactly-once for Kafka-to-Kafka stream processing, and nothing for the external side effect.

Ordering, partitions, and backpressure

Kafka orders records within a partition, never across partitions. You buy parallelism by adding partitions and consumers (one partition maps to at most one consumer in a group), and you pay for it with weaker global order and costlier rebalances. Key by entity so one customer's events share a partition and stay ordered while different customers run in parallel. One slow or failing message head-of-line-blocks its partition until it clears or hits the DLQ.

Backpressure is the other failure mode. A consumer slower than the producer grows the queue without bound until disk or memory dies. Levers: scale consumers up to the partition count, bound the queue and reject or slow producers (429), or shed load. Consumer lag (offset lag, or queue depth in RabbitMQ) is the number you alert on.

The transactional outbox

The dual-write problem: "save to the database" and "publish to the broker" are two systems with no shared transaction, so a crash between them either drops the event or emits one for a write that rolled back.

BEGIN;
INSERT INTO orders (...) VALUES (...);
INSERT INTO outbox (topic, payload) VALUES ('order.placed', ...);
COMMIT; -- one atomic commit: both land or neither does

A relay (a poller, or CDC via Debezium reading the write-ahead log) then publishes rows from the outbox and marks them sent. The relay can crash after publishing before marking sent, so the outbox is at-least-once and the consumer still dedups. The outbox fixes atomicity of the write and the event, not duplicate suppression downstream.