In June I read The Chubby Lock Service for Loosely-Coupled Distributed Systems. I wanted to understand why a system would expose a file-and-lock interface on top of consensus instead of asking every application team to implement leader election itself.
Chubby is designed for coarse-grained coordination and small, reliable metadata—not high-volume data storage or fine-grained transaction locking.
Paper: The Chubby Lock Service for Loosely-Coupled Distributed Systems (PDF)
A cell is a replicated state machine
A Chubby deployment is called a cell, commonly with five replicas. One replica is master for a lease period. The replicas use Paxos to keep the service state consistent and elect a new master after failure.
Five replicas can continue with any three available. The system tolerates minority failures while preserving one ordered history of mutations.
Clients discover the master and send operations to it. Replicas are not a general read-scaling layer; the service prioritizes simple semantics and reliability over throughput.
Why expose locks and files
The namespace resembles a small file system. Clients open named nodes, read and write small values, acquire advisory locks and receive change notifications.
This interface supports common coordination patterns:
- Elect a primary by acquiring a named lock.
- Publish the current primary’s address in the associated file.
- Store a small bootstrap pointer or configuration value.
- Detect membership through lock ownership and sessions.
- Notify clients when metadata changes.
A familiar API also reduced the temptation for teams to build ad hoc consensus protocols.
Coarse-grained, not fine-grained locking
Chubby locks may be held for hours or days. They protect roles such as “active master,” not every row update in a database.
Using a remote consensus-backed service for fine-grained locks would add latency, load and a large dependency surface to ordinary operations. Applications should keep their data-plane concurrency local or inside the database and use Chubby for infrequent control-plane decisions.
This boundary is one of the strongest lessons in the paper:
coordination plane → small state, strong agreement, low update ratedata plane → large state, high throughput, application-specific pathsSessions connect locks to liveness
A client maintains a session with the master through keepalives. Locks and handles are associated with the session. If the session expires, the service can release locks and let another client progress.
The client cannot use one missed keepalive as proof that its session expired. Network delay may separate the client’s view from the master’s. The protocol therefore includes lease intervals and a grace period around master failover.
This is subtle: a lock is safe only when ownership and liveness are tied to a server-authoritative lease, not the client’s local belief.
Master leases and failover
The master remains master while its lease is valid. Replicas will not elect a conflicting master during that period. After a failure, a new master waits until it is safe to assume the old lease has expired, reconstructs sessions and resumes keepalives.
This may delay availability during failover, but it protects safety. Coordination systems commonly trade a short pause for avoiding two active masters.
Stale lock holders and sequencers
Imagine client A holds a lock, pauses for a long time and loses its session. Client B acquires the lock and writes to an external storage system. Client A wakes up and still believes it owns the lock. An advisory lock alone cannot stop A from performing a stale write.
Chubby provides a sequencer containing lock-generation information. A protected service can reject operations carrying an old sequencer.
A acquires generation 8A pauses; lease expiresB acquires generation 9storage rejects A's generation 8 requestThis is the fencing-token pattern. It closes the gap between “the lock service changed ownership” and “the resource knows ownership changed.”
Without fencing, a distributed lock can create confidence without safety.
Caching and invalidation
Chubby clients cache file data and metadata to reduce server load. The master sends invalidations when cached state changes. A mutation that affects cached data waits for invalidations to be acknowledged or for relevant sessions to expire.
Caching makes a low-throughput consensus service useful to many clients, but invalidation fan-out becomes part of write latency. The paper describes how unexpected client behaviour forced refinements to notification and caching mechanisms.
This is a useful reminder that production systems are shaped by how APIs are actually used, not only by their intended use.
Events are hints, not a durable queue
Clients can receive events for file changes, lock releases and session state. Notifications help avoid polling, but clients must still reread authoritative state after reconnecting or missing an event.
Treating notifications as hints leads to robust logic:
notification → something may have changed → read current stateTreating every notification as an exactly-once durable event stream would give Chubby a responsibility it was not designed to carry.
Why Chubby improves other systems
Google systems use Chubby as a stable root. GFS can select a master. Bigtable can elect its master, discover tablet servers and store the root tablet location.
The availability benefit does not come from making those systems incapable of progress without Chubby forever. Clients cache metadata, sessions absorb short interruptions and the dependency is kept out of high-volume paths.
Operational lessons
The paper is unusually valuable because it describes misuse and evolution:
- Clients created too many files because the interface looked like a file system.
- Some applications polled instead of using caching and events.
- Retry storms could amplify an outage.
- Developers expected stronger availability than a five-replica cell could provide across broad failures.
Backoff, request batching, quotas and good client libraries are part of correctness at scale. A sound consensus core can still be overwhelmed by poorly behaved clients.
What I keep from the paper
Chubby turns consensus into a small reusable service with understandable semantics. Its design works because it keeps a narrow scope.
The questions I keep are:
- Is this coordination state small and infrequently updated?
- What is the authoritative lease, and how does it expire?
- Does the protected resource validate fencing tokens?
- Can clients tolerate temporary unavailability of coordination?
- Are notifications treated as hints followed by a read?
- What prevents retries and polling from overloading recovery?
The biggest lesson is not “use a distributed lock.” It is to make ownership externally verifiable and to keep strongly coordinated state away from the data plane.