Event-driven architecture

Events vs commands

An event is a fact about something that already happened, named in the past tense: OrderPlaced, SeatsReserved, ReservationCanceled. It is immutable, and because it already happened, a subscriber cannot reject it. Events go out pub/sub, so many services react to one event and the publisher never knows who is listening.

A command is intent, a request to do something, named imperatively: ReserveSeats, ChargePayment. It is directed at a handler that can validate it and reject it. Model the business action rather than the row change: send BookHotelRoom instead of SetReservationStatusToReserved.

The dual-write problem and the outbox

Most handlers do two things at once: update the database and publish an event. Run those as two separate operations and a crash in between either commits the row without publishing (the event is lost) or publishes and then rolls the row back (a phantom event for data that does not exist). You cannot enlist a database and a broker in one transaction, so two independent writes eventually disagree.

The transactional outbox closes the gap. In the same local transaction that writes the order, insert the event into an outbox table. The event persists if and only if the transaction commits. A separate relay reads the outbox and publishes each row, so the publish is driven by committed state.

Why consumers must be idempotent

The relay can publish a row and then crash before it records that the row was sent, so on restart it publishes it again. Delivery is at-least-once, so every consumer sometimes sees the same event twice. Make a duplicate a no-op: keep a processed_messages table keyed on (consumerId, messageId), insert the id inside the same transaction as the business update, and let the primary-key conflict discard the repeat.

Wrong "We set enable.idempotence=true, so a message can't be handled twice." That flag only drops duplicate appends from a producer retrying to a partition. Right delivery to a consumer is still at-least-once, so a rebalance or redelivery hands you the same event again, and the consumer needs its own dedup key.

What CQRS actually requires

CQRS splits the write model that handles commands from the read model that serves queries, so each is shaped for its own load. It does not require two databases: the first level is separate models over one store. It also does not require event sourcing, which is a common pairing rather than a dependency. When the read model is a separate store kept current by events, it is eventually consistent with the writes, so a user can query data that does not yet reflect the command they just sent.

Publishing the outbox: polling vs log tailing

Two mechanisms move outbox rows to the broker. A polling publisher selects unpublished rows, sends them, then marks them published or deletes them. It runs on any SQL database, but keeping events in order is tricky.

Log tailing reads the commit log instead: the Postgres write-ahead log, the MySQL binlog, or DynamoDB streams. Debezium tails the log as a Kafka Connect connector and emits one change event per inserted outbox row. Its Outbox Event Router uses the aggregateid column as the Kafka message key, which pins every event for one entity to a single partition and keeps them ordered. It is database-specific and can re-emit a row after a crash. Outbox rows are insert-only; the router treats the table as a queue and rejects updates.

When event sourcing is not worth it

Event sourcing stores an entity as an append-only sequence of state-changing events and rebuilds state by replaying them. It removes dual-write, since each change is a single append, and gives an audit log. It is also a specialized, expensive pattern; Microsoft's own guidance is that for most systems, traditional data management is enough.

The costs are concrete. You cannot query current state with SQL, so you need CQRS and materialized projections that get rebuilt when they change. Events are immutable, so schema changes go through tolerant readers, versioned event types, or upcasting old events at read time, never by rewriting history; a mistake is undone with a compensating event. Right-to-be-forgotten fights an append-only store, so PII lives outside it or is crypto-shredded by destroying a per-subject key. Reach for the pattern on a payment ledger or an order pipeline rather than across a whole system.

Wrong "We can use Kafka as the event store." A broker lacks per-entity stream queries and optimistic concurrency, so it is a distribution layer, not a system of record. Snapshots do not change that; a snapshot shortens replay while the stream stays the source of truth.

Sagas give you ACD, not ACID

A saga holds consistency across services as a sequence of local transactions, each updating one database and then publishing an event (choreography) or acting on a command from an orchestrator. There is no distributed transaction and no automatic rollback: you write compensating transactions that explicitly undo earlier steps when a later one fails.

The property it surrenders is isolation. Concurrent sagas can observe each other's intermediate writes and produce anomalies unless you add countermeasures. Each step still updates a database and publishes a message, the same dual-write, so it needs the outbox or event sourcing underneath.

Schema evolution and compatibility direction

The Confluent Schema Registry defaults to BACKWARD compatibility, checked against the previous version only. BACKWARD lets a new schema read old data: add optional fields, delete fields, upgrade consumers first. FORWARD is the mirror, an old schema reading new data, so upgrade producers first. FULL permits only adding or removing optional fields, and both sides upgrade in either order. Transitive variants check every prior version. Under FORWARD and FULL you can only remove a field that was optional or had a default; BACKWARD lets you remove any field.