Reading ZooKeeper: a kernel for coordination

In August I read ZooKeeper: Wait-free Coordination for Internet-scale Systems. I wanted to understand the phrase “coordination kernel”: instead of placing every recipe inside the server, ZooKeeper exposes a small set of ordered data operations from which clients build recipes.

Paper: ZooKeeper: Wait-free Coordination for Internet-scale Systems (PDF)

Open full size ↗

The server stores a small hierarchy

ZooKeeper presents a namespace of znodes resembling a file system. Each znode stores a small value and metadata, and can have children.

The namespace is intended for coordination data:

• Configuration.
• Membership.
• Leader identity.
• Barriers and queues.
• Lock contenders.

It is not a general file system or large-object store. Keeping the dataset small allows servers to hold it in memory while persisting updates through a log and snapshots.

Persistent, ephemeral and sequential znodes

Znode creation modes provide the pieces for higher-level recipes.

Persistent znodes remain until explicitly deleted. They fit configuration and stable naming.

Ephemeral znodes are tied to a client session and disappear when the session expires. They fit membership and ownership announcements.

Sequential znodes receive a monotonically increasing suffix. They give contenders an order without requiring every client to choose a unique number.

Combining ephemeral and sequential nodes makes a useful lock recipe: each contender creates a node, sorts contenders and watches the predecessor.

Ordered writes through atomic broadcast

An ensemble has a leader and followers. Writes are transformed into transactions and ordered through the ZooKeeper Atomic Broadcast protocol. A quorum persists an update before it becomes committed.

All replicas apply committed writes in the same order. This gives linearizable writes and one system-wide update history.

Reads can be served locally by the connected server, which gives high read throughput but may return older state. A sync operation can bring the client’s view up to date before a read when needed.

The system therefore distinguishes:

ordered mutation → quorum pathordinary read    → local replica path

That split is the reason a read-heavy coordination service can scale beyond one leader’s read capacity.

The two ordering guarantees

The paper emphasizes two guarantees:

• Linearizable writes: mutations appear in one global order.
• FIFO client order: operations from one client execute in submission order.

These guarantees compose into useful recipes. They do not mean every read is linearizable by default. The client must understand which operation establishes the freshness boundary it needs.

Sessions model client liveness

A client connects to one server and maintains a session through heartbeats. If that server fails, the client can reconnect to another server while the session remains valid.

If the ensemble does not hear from the client before the negotiated timeout, the session expires and ephemeral znodes are deleted.

As with any lease-like mechanism, a disconnected client cannot decide by itself that it still owns an ephemeral role. It should stop acting when it loses the session and re-establish state only after a successful reconnect.

Watches avoid constant polling

A client can place a watch while reading a znode or its children. When relevant state changes, ZooKeeper sends a notification.

Watches are one-shot triggers. After notification, the client reads current state and installs another watch. Several changes can collapse into one notification, so a watch is not a change log.

Correct pattern:

read state + set watch→ receive notification→ read current state + set watch again

The ordering guarantee ensures the notification is observed before the client sees later state on that session, making cache invalidation practical.

Building a lock without a herd

A naive lock implementation makes every contender watch the lock node. When it is released, all clients wake up, but only one can win. This is the herd effect.

The sequential-node recipe is better:

1. Create an ephemeral sequential node under /locks/job.
2. List and sort the children.
3. If mine is smallest, I hold the lock.
4. Otherwise watch only my immediate predecessor.
5. When it disappears, repeat the check.

Only the next contender wakes for each release. The queue is visible and recoverable from ZooKeeper state.

As with Chubby, a protected external service may still need a fencing token. Session expiry prevents future ownership but cannot physically stop a paused old process from issuing a stale request.

Leader election and membership

Leader election uses a similar recipe. Candidates create ephemeral sequential nodes; the smallest sequence becomes leader. Ephemeral lifetime removes a failed candidate, and predecessor watches avoid waking everyone.

Membership can be represented by ephemeral children:

/workers/worker-001/workers/worker-002/workers/worker-003

Consumers watch /workers and reread the child list after a change. The resulting view is derived from session-backed ownership rather than periodic application cleanup.

What “wait-free” means in the paper

The term does not mean ZooKeeper uses no waiting internally. It describes an API direction: operations manipulate simple data objects rather than placing blocking coordination primitives inside the server.

Slow or failed clients should not hold a server-side operation open and block unrelated clients. Higher-level blocking behaviour is built by clients using state and watches.

This keeps the service core small and lets different applications build different coordination recipes.

Failure and recovery boundaries

A quorum must be available to commit writes. During leader election, writes pause. Reads from connected replicas may continue depending on server state, but clients must not interpret availability as freshness.

The replicated log and snapshots recover server state. Sessions require careful handling across reconnects, and clients must treat CONNECTION_LOSS as an uncertain result: a create may have committed even if the reply was lost.

Sequential node names or client-provided identifiers can make retries discoverable rather than blindly duplicating an operation.

What I keep from the paper

ZooKeeper provides a small ordered substrate, not a catalogue of magical distributed primitives.

I keep these rules:

• Store coordination metadata, not application bulk data.
• Use ephemeral state for session-owned membership.
• Use sequential nodes to establish order.
• Watch one predecessor to avoid herd effects.
• Treat watches as invalidations, then reread.
• Distinguish local read availability from freshness.
• Make ambiguous retries discoverable or idempotent.

The paper’s lasting idea is that a carefully chosen kernel can support many coordination patterns while keeping consensus and recovery in one professionally operated place.