Reading Kafka: the log as the system

In January I read Kafka: a Distributed Messaging System for Log Processing. I expected a message broker design. The paper instead starts from a durable, partitioned log and makes both online and offline consumers read from it at their own pace.

Paper: Kafka: a Distributed Messaging System for Log Processing (PDF)

The workload was bigger than traditional messaging

LinkedIn needed to collect activity events and operational logs at high volume. Data had to feed both low-latency applications and offline analytics.

Traditional enterprise brokers offered rich per-message delivery semantics, but that bookkeeping was expensive. Log collectors handled throughput, but did not provide the consumption model needed by many independent applications.

Kafka’s early design combines three ideas:

append-only log       → efficient storagemessaging API         → producers and consumers are decoupledconsumer-owned offset → replay without broker delivery state

Topic partitions are independent logs

A topic is divided into partitions. Each partition is an ordered, append-only sequence stored by one broker in the original paper.

Messages receive monotonically increasing offsets within a partition. The offset is a logical position, not a global event identifier. Ordering exists inside one partition; there is no total ordering across an entire topic.

Partitioning provides parallelism. Producers can spread records across brokers, and consumers can process partitions concurrently. The partition key therefore controls both ordering and load.

If all events for an account must stay ordered, they should use a stable account-derived key. A hot account can then become a hot partition. Ordering and balancing are connected trade-offs.

The broker stores simple segment files

Kafka appends messages to partition log segments. Sequential appends work well with disks and the operating system page cache. Old segments are deleted according to retention policy rather than individual acknowledgements.

This is a major simplification. The broker does not need to maintain a deletion state for every message and consumer. The same retained bytes can serve many consumers, and a consumer can replay old data by moving its offset backwards.

The log is both the queue and the archive for the configured retention period.

Consumers pull and track position

Instead of the broker pushing one message and waiting for an acknowledgement, a Kafka consumer asks for a range beginning at an offset.

fetch(partition=3, offset=840, maxBytes=...)

The broker can return a batch of sequential bytes. The consumer advances its position after processing.

Keeping the offset with the consumer makes failure behaviour explicit:

Commit before processing: a crash can lose work.
Commit after processing: a crash can repeat work.
Store offset with the output: processing can be made atomic within the destination system.

Kafka does not remove the need for idempotency. It gives the application a durable position from which to reason about retries.

Consumers sharing a group divide partitions so one group member processes a partition at a time. Different groups read the same topic independently.

one group, many members → parallel queue-like consumptionmany groups             → publish-subscribe fan-out

Maximum useful parallelism for a group is bounded by partition count. Adding consumers beyond that leaves some idle. Adding partitions later increases parallelism but changes key-to-partition mapping unless routing is controlled.

Batching is part of the design

High throughput comes from amortising fixed costs:

Producers batch multiple messages per request.
Brokers append batches sequentially.
Consumers fetch batches.
The network sends larger buffers instead of many tiny packets.

Batching improves throughput but adds latency while a batch fills. Real systems need limits by both size and time so low traffic does not wait indefinitely.

Page cache and zero-copy

The paper leans on the operating system rather than building a separate application cache. Recently written log data is already in the page cache. Sending file-backed data with sendfile can avoid copying bytes through application buffers.

Conceptually:

ordinary path: disk/page cache → kernel → user buffer → kernel socketzero-copy path: disk/page cache → kernel socket

The exact hardware path is more nuanced, but the design principle is useful: a simple on-disk representation can align storage and network transfer.

Retention changes the meaning of consumption

Traditional queues often remove a message after successful delivery. Kafka retains by time or size, independent of which consumer has read it.

This enables:

Rebuilding a derived index.
Adding a new consumer without changing producers.
Replaying after a bug fix.
Comparing two versions of a processor.
Feeding both streaming and batch systems.

Retention also turns disk capacity and recovery time into design parameters. A seven-day log is useful only if a consumer can catch up before data expires.

Historical boundary: the 2011 paper is not modern Kafka

The original paper describes an early system. It explicitly notes limitations around replication and automatic broker balancing. Modern Kafka added replicated partitions, stronger producer acknowledgements, idempotent producers, transactions, richer group coordination and later KRaft metadata management.

I keep the layers separate:

original insight → partitioned persistent log + pull + consumer offsetslater evolution  → replication, consensus, transactions and operations at scale

Reading modern guarantees back into the early paper makes its measured simplicity hard to see.

Failure reasoning

For a producer retry, duplicate appends are possible unless the protocol assigns stable identities and sequence numbers. For a consumer retry, duplicate processing is possible unless output is idempotent or committed atomically with the offset.

With replicated modern Kafka, a partition leader orders writes and followers copy the log. Availability and durability depend on replica placement, in-sync replica rules and acknowledgement configuration. Those are later mechanisms layered around the same ordered partition log.

What I keep from the paper

Kafka’s core move is to make the durable log the source of truth and let consumers manage their position. That removes per-consumer delivery state from the broker and makes replay normal.

The questions I keep are:

What key determines ordering, and can it become hot?
Is the consumer’s output idempotent under replay?
Where is the offset stored relative to the side effect?
Can consumers catch up within retention?
Is the workload throughput-bound or latency-bound?
Which guarantees come from the original log model and which require modern replication features?

Once those are clear, topics, partitions and offsets stop being vocabulary and become a concrete data-flow model.