Kafka internals

A topic is split into ordered partition logs

A Kafka topic is a named stream of records. Kafka divides that stream into partitions. Each partition is a log whose records have offsets that identify positions in that partition.

Kafka's ordering guarantee belongs to one partition. It does not provide one total order across every partition in a topic. The producer chooses the destination partition. It can use a record key for semantic partitioning, such as sending every event for one customer to the same partition.

topic: customer-events

partition 0: offset 0, offset 1, offset 2
partition 1: offset 0, offset 1

The two offset sequences are independent. There is no meaningful comparison between partition 0 offset 2 and partition 1 offset 1 that establishes which record comes first for the whole topic.

Wrong "Kafka preserves the order of every record in a topic."

Right Kafka preserves order within a partition. Records that require a shared order need a partitioning rule that places them in the same partition.

Partitions also set a scaling boundary. A traditional consumer group assigns each partition to one consumer in that group at a time. More partitions can therefore allow more group members to process in parallel, but they divide the data into more independent ordered logs.

Consumer groups share partitions

Consumers with the same group.id form a consumer group. The group distributes subscribed partitions among its members. Different groups consume the topic independently, so two applications can each read the same topic using different group IDs.

If a group has four partitions and two consumers, each consumer can receive a subset of the four. If the group has more consumers than partitions, some consumers cannot receive a partition under the traditional consumer-group model.

Membership and subscriptions can change. Kafka then performs a rebalance to produce a new partition assignment. A process can leave, fail, or join. An administrator can also add partitions to a subscribed topic. Each of these can lead to reassignment.

Kafka 4.3 supports both the Classic and Consumer group protocols. The newer Consumer protocol became generally available in Kafka 4.0 and uses incremental rebalancing. It is available on the server, but a client enables it with this configuration:

group.protocol=consumer

The Classic protocol remains the default in the Kafka 4.3 consumer configuration. Under the Consumer protocol, the broker controls the heartbeat interval, session timeout, and server-side assignment strategy.

Offsets record group progress

A consumer fetches records from a position in each assigned partition. A committed offset is the position the group will use after a restart or rebalance. Kafka's Java consumer documentation defines the committed value as the next record the application will consume.

After processing offset 41, the next offset is 42:

OffsetAndMetadata next = new OffsetAndMetadata(42L);
consumer.commitSync(Map.of(topicPartition, next));

Committing 41 does not mean "offset 41 is done." It tells the next consumer to start at 41, so that record can be delivered again.

Automatic commits periodically commit offsets in the background when enable.auto.commit=true. Manual commits let the application choose the point at which progress is recorded. Neither mechanism proves that an external side effect succeeded. The timing relative to processing determines the failure window.

Wrong "Committing an offset deletes the record from Kafka."

Right An offset commit records a consumer group's position. Kafka retains records according to the topic's retention policy, independently of whether a group consumed them.

Replication keeps partition logs available

Kafka replicates each partition across a configured number of brokers. One replica is the leader. Followers copy the leader's log, and writes go to the leader.

The in-sync replica set, usually shortened to ISR, contains replicas that are caught up enough to participate in Kafka's committed-message guarantee. The replication factor is the configured number of replicas. The ISR is the current set of replicas considered in sync. These numbers can differ when a follower falls behind or becomes unavailable.

With acks=all, every replica currently in the ISR must acknowledge the write. A topic's min.insync.replicas can require a minimum ISR size before such a write succeeds.

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 6 --replication-factor 3 \
  --config min.insync.replicas=2

With this topic policy and acks=all, an ISR of two can accept a write. An ISR of one cannot. The broker rejects the write instead of lowering the configured durability floor.

Retention is separate from consumption

The default topic cleanup policy is delete. Kafka removes old log segments when their configured time or size limit is reached. Cleanup works on segments, so expiration does not imply immediate deletion of an individual record at an exact age.

The compact policy has a different purpose. It retains at least the latest value for each key in a partition. Compaction runs in the background, removes superseded records, preserves the order of retained records, and does not change their offsets. A keyed record with a null value is a tombstone that marks the key for deletion.

Kafka can apply both policies:

cleanup.policy=delete,compact

In that case, time or size retention can discard old segments while compaction removes superseded keyed records from the segments that remain.

Ordering depends on partition assignment

Kafka gives each partition its own ordered offset sequence. The producer controls the partition choice, either directly or through a partitioning function. A key supports semantic partitioning: records for the same entity can be sent to one partition so a consumer sees that entity's events in partition order.

This guarantee has two boundaries. First, Kafka does not merge several partitions into one global order. Second, producer retry settings can affect record order. Kafka 4.3 warns that retries may reorder batches for one partition when idempotence is disabled and max.in.flight.requests.per.connection is greater than one. A later batch can succeed before an earlier failed batch is retried.

The current producer defaults enable idempotence when its required settings do not conflict. An explicit safe configuration makes the intended contract visible:

enable.idempotence=true
acks=all
retries=2147483647
max.in.flight.requests.per.connection=5

Kafka requires idempotence to use acks=all, retries greater than zero, and at most five in-flight requests per connection. Idempotence prevents duplicate writes caused by producer retries within a producer session and preserves ordering under those constraints. It does not make a consumer's external side effects exactly once.

Wrong "A stable key gives global ordering because all consumers agree on the key."

Right A stable key can place related records in one partition. The ordering guarantee remains local to that partition.

Rebalancing transfers ownership

A consumer group gives one member ownership of each assigned partition. Assignment changes when group membership or subscriptions change, and can also change when the topic gains partitions. The consumer then resumes from the group's committed position for a newly assigned partition.

This is why offset handling and rebalancing cannot be designed separately. Kafka's ConsumerRebalanceListener gives an application callbacks for revoked, assigned, and lost partitions. The Java API recommends committing offsets for revoked partitions to reduce duplicate processing when ownership changes.

consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
  public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
    consumer.commitSync(nextOffsetsFor(partitions));
  }

  public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
    // Initialize partition-local state for the new assignment.
  }
});

The committed value for each partition must be the next offset to process. If offset 41 finished, commit 42. A commit for an offset beyond completed work can skip records after failure. A commit behind completed work can repeat records.

Long processing can also cause reassignment. max.poll.interval.ms bounds the delay between poll() calls under group management. If the consumer exceeds it, the group considers the member failed and rebalances. Reducing max.poll.records can make each returned batch smaller without changing the consumer's underlying fetch behavior:

max.poll.interval.ms=300000
max.poll.records=200

Kafka 4.x has two group protocols. Classic groups use client-side assignment settings and eager or cooperative assignors. Consumer groups using the newer protocol use a fully incremental design with server-side assignment. In Kafka 4.3, the client still has to opt in with group.protocol=consumer; classic is the documented default.

Commit timing creates the delivery behavior

Consider a consumer that writes to an external database.

Process first, then commit:

poll -> database write -> commit next offset

If the process crashes after the database write and before the commit, the replacement consumer starts from an older committed offset. It can repeat the write. This is at-least-once processing and requires the destination to tolerate or deduplicate repeats.

Commit first, then process:

poll -> commit next offset -> database write

A crash after the commit and before the database write makes the group resume after an unprocessed record. This is at-most-once processing, where records may be lost to the application.

Wrong "commitSync provides exactly-once processing because it waits for Kafka to acknowledge the commit."

Right commitSync confirms the Kafka offset update or reports an error. It cannot atomically confirm an unrelated database write.

commitAsync does not block. Errors go to its callback when one is supplied and are otherwise discarded. Kafka sends asynchronous offset commits in invocation order and calls their callbacks in that order. A later commitSync waits for earlier asynchronous commit callbacks before it returns. These API ordering guarantees do not remove the processing-versus-commit failure window.

ISR is a live set, not the replica count

Each partition has one leader and zero or more followers. The replication factor counts all configured replicas. ISR membership reflects which replicas are currently in sync.

For acks=all, Kafka requires every current ISR member to acknowledge. min.insync.replicas adds a lower bound on how many ISR members must exist for the write to succeed. With replication factor 3 and min.insync.replicas=2, all three acknowledge while all three remain in the ISR. If one drops out, the two remaining ISR members can acknowledge. If another drops out, the write fails.

acks=1 only waits for the leader and does not activate the min.insync.replicas protection used by acks=all. A replication factor of three therefore does not, by itself, state what the producer waits for before treating a write as successful.

Delete retention and compaction solve different problems

cleanup.policy=delete discards old segments when a time or size limit is reached. cleanup.policy=compact retains at least the latest value for each key. A topic can use both policies.

Compaction is not immediate and does not reduce the log to exactly one record per key at every moment. The active head contains recent records, and background cleaners process eligible segments. Retained records keep their original offsets, so compaction can leave gaps. Reading from an offset whose record was removed starts at the next higher retained offset.

before: offset 7 key=A old
        offset 8 key=B value
        offset 9 key=A new

after:  offset 8 key=B value
        offset 9 key=A new

A null value for a key is a tombstone. It causes the older value to be removed, and the tombstone can itself be removed after delete.retention.ms. A consumer rebuilding state from the beginning must catch up within that tombstone-retention bound if it must observe every deletion marker.

The partition is the unit of ordering and replication

Kafka's core unit is a replicated partition log. The producer sends a record to the leader for one partition. Followers copy that leader's offsets, records, and ordering. Consumer-group assignment also operates on partitions, which ties ordering, processing parallelism, state locality, and failover to the same boundary.

Choosing an entity key can preserve per-entity order by routing that entity to one partition. It also concentrates that entity's traffic and state on one partition. Adding partitions increases the number of logs that can be distributed, but it cannot create a topic-wide order.

Producer ordering needs one more qualification. Kafka 4.3 documents a reordering risk when retries are enabled, idempotence is disabled, and more than one request is in flight per connection. If the earlier of two batches fails while the later batch succeeds, retrying the earlier batch can append it second.

enable.idempotence=true
acks=all
retries=2147483647
max.in.flight.requests.per.connection=5

Those values satisfy the Kafka 4.3 idempotence constraints. Idempotence handles duplicate sends from producer retries. A transactional.id extends the producer API to transactions and supports atomic writes across partitions plus consumer-offset updates.

Wrong "Idempotent production means the whole application processes each business event exactly once."

Right Producer idempotence addresses retry duplicates in Kafka writes. End-to-end processing also includes consumer progress and every destination side effect.

Rebalance safety lives at the ownership boundary

A rebalance changes which member owns a partition. Any partition-local buffer, open transaction, or offset tracker must follow that ownership change. Kafka's rebalance listener distinguishes revocation from loss. Revocation gives the current consumer an opportunity to commit or clean up before transfer. Loss means the partitions may already belong to another consumer, so writing offsets or output as the old owner can be unsafe.

For manual offset management, track completed progress per partition and commit only those next offsets:

Map<TopicPartition, OffsetAndMetadata> completed = new HashMap<>();

for (ConsumerRecord<String, Order> record : records) {
  process(record);
  completed.put(
      new TopicPartition(record.topic(), record.partition()),
      new OffsetAndMetadata(record.offset() + 1)
  );
}

consumer.commitSync(completed);

The example processes records synchronously. Parallel processing within one partition needs an additional completion tracker so a later finished record cannot advance the committed offset past earlier unfinished work.

max.poll.interval.ms is another ownership boundary. Exceeding it causes the group to treat the consumer as failed and reassign partitions. Static membership with group.instance.id changes the timing: on a poll-interval breach, a static member stops heartbeating and reassignment waits until the session timeout. This can avoid rebalances during brief restarts, but it delays replacement of a failed instance.

As of Kafka 4.3, the newer Consumer rebalance protocol is available but remains client opt-in through group.protocol=consumer. It became generally available in Kafka 4.0. The broker controls its heartbeat interval, session timeout, and server-side assignor. Its fully incremental design removes the Classic protocol's global synchronization barrier. A version-neutral answer that says every rebalance revokes every partition is therefore too broad.

ISR policy trades write availability for durability

The leader orders writes. Followers fetch and copy that log. Kafka considers a record committed only after it reaches every replica in the current ISR and the ISR satisfies min.insync.replicas. Consumers do not receive the record before those conditions are met.

With this policy:

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 6 --replication-factor 3 \
  --config min.insync.replicas=2

and this producer setting:

acks=all

all three replicas acknowledge while all three are in the ISR. Two acknowledge after one falls out of the ISR. With only one ISR member, the write fails with a not-enough-replicas error. Availability falls before the final in-sync copy is used for acknowledged writes because the configured durability floor is two.

Wrong "acks=all means a fixed quorum of all configured replicas."

Right It means every replica currently in the ISR, subject to the min.insync.replicas lower bound. The replication factor, ISR membership, acknowledgements, and minimum ISR are separate parts of the policy.

Leader election has a newer case in Kafka 4.3. Eligible Leader Replicas (ELR) can track replicas outside the ISR that remain safe to elect under the strict min.insync.replicas rule. ELR is enabled by default on new clusters starting with Kafka 4.1. When ELR is enabled, the controller prefers an ISR member, then an unfenced ELR member, then an unfenced last known leader. This is why the Kafka 4.3 configuration reference warns that ELR changes the semantics around min.insync.replicas.

The separate topic setting unclean.leader.election.enable permits election of a replica outside the ISR as a last resort and may cause data loss. Enabling unclean election chooses availability over the safety of a normal leader election.

Exactly once has a transaction boundary

Kafka documents at-least-once delivery as the default. A consumer can implement at-most-once behavior by committing before processing, at the cost of losing application processing when failure lands between those operations.

For a Kafka-to-Kafka read, process, and write flow, the transactional producer can atomically publish output records and update the consumed offsets. Consumers that need to hide aborted transactional records use read_committed:

enable.auto.commit=false
isolation.level=read_committed

transactional.id=orders-transformer-1

read_committed returns committed transactional records and all non-transactional records. It withholds records at or beyond the last stable offset while an earlier transaction remains open. It does not turn a consumer into a transactional writer.

Exactly-once output to another system needs cooperation from that destination. Kafka's design documentation gives the general requirement: store the consumer position in the same place as the output, or otherwise coordinate the two updates. A Kafka transaction cannot atomically include an arbitrary HTTP request or database operation merely because the input came from Kafka.

Compaction preserves state, offsets, and order

Compaction processes eligible log segments in the background. For each key, Kafka retains at least the latest value. It can temporarily retain several values for a key, since recent data and segments awaiting cleaning remain uncompacted.

Compaction removes records without renumbering the survivors. Offsets stay permanent positions and ordering is preserved. A compacted log can therefore have gaps, and seeking to a removed offset resolves to the next higher retained record.

The lag settings describe eligibility, not an exact cleaning deadline:

cleanup.policy=compact
min.compaction.lag.ms=60000
max.compaction.lag.ms=3600000

min.compaction.lag.ms gives a lower bound on how long a record stays in the uncompacted head. max.compaction.lag.ms bounds when a record becomes eligible for compaction, while the documentation warns that cleaner availability and cleaning time mean the compaction deadline is not hard.

A tombstone is a keyed record with a null value. Consumers reconstructing a state snapshot must account for delete.retention.ms, because Kafka can remove tombstones after that interval. With cleanup.policy=delete,compact, even the latest value for a key can eventually disappear when time or size retention removes its segment.