In March I read SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol. I wanted a more precise model for “gossip” than nodes randomly telling each other everything.
SWIM combines two responsibilities that are related but should be designed separately: detecting an unresponsive member and spreading that observation through the group.
Paper: SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol (PDF)
Why all-to-all heartbeats stop scaling
If every node sends a heartbeat to every other node, each member sends and receives work proportional to the cluster size. Across the whole cluster, traffic grows quadratically.
Centralised monitoring reduces peer traffic but creates a bottleneck and a special failure domain. A membership protocol should keep per-node work roughly constant while still detecting and spreading failures quickly.
SWIM’s answer is randomized probing plus infection-style dissemination.
Separate failure detection from dissemination
The protocol has two conceptual components:
failure detector → produce a local suspicion about a memberdissemination → spread membership updates through the clusterThis separation makes the system easier to tune. Probe frequency affects detection time. Gossip fan-out and piggybacking affect convergence. Improving one does not require redesigning the other.
Direct probing
During each protocol period, a node selects a random member and sends PING. If the target replies with ACK before the timeout, the probe succeeds.
Random selection spreads checks across the membership. Over several periods, every member is likely to be probed without each node maintaining a heartbeat stream to every peer.
The timeout is a trade-off. A short timeout detects failure quickly but mistakes delay for death. A long timeout reduces false positives but slows recovery decisions.
Indirect probing
When a direct ping times out, the probing node asks several other members to send PING-REQ to the target. Those helpers probe the target and relay an acknowledgement.
A ──PING──> B direct path failsA ──PING-REQ──> C ──PING──> BA <────── relayed ACK ───── CIndirect probing distinguishes a failed target from a broken path between two healthy nodes. It adds a small amount of traffic only on the suspected failure path.
This is a general technique: use independent observers before converting one local communication failure into a cluster-wide decision.
Suspicion before removal
Immediately declaring a node dead after one failed probe creates false positives under packet loss, pauses or overload. SWIM can first mark the node suspect and disseminate that state.
If the node is alive, it can refute suspicion by announcing a newer incarnation. If suspicion remains long enough, members transition it to failed.
The state progression is easier to operate than a boolean:
alive → suspect → failed ↑ refute with newer incarnationIncarnation numbers prevent an old failure message from overriding newer evidence that the process is alive.
Infection-style dissemination
Membership updates are piggybacked on ordinary protocol messages. A node that learns an update includes it in several later messages. Recipients repeat the process.
The update spreads probabilistically, like an epidemic. No single message has to reach every node. Redundant paths make dissemination resilient to packet loss.
Old updates cannot be piggybacked forever. Implementations bound retransmission based on cluster size and favour recent or important changes. Full-state synchronization or periodic repair may still be useful when a member misses many updates.
Local views are allowed to differ
SWIM is weakly consistent. At a given instant, two members may disagree about whether another node is alive or suspect.
That is not necessarily a correctness bug. Membership is often an input to routing or replica repair, and the application must tolerate temporary disagreement.
The dangerous mistake is to treat failure-detector output as perfect truth. In an asynchronous system, a timeout proves only that a response did not arrive in time.
What “scalable” means here
SWIM aims for constant expected message load per member as the group grows, with detection time that grows slowly. Random probing prevents one monitor from carrying all the work, and piggybacking reuses messages already being sent.
There are still practical limits:
- Large membership lists consume memory.
- NAT, firewalls and asymmetric routing complicate probing.
- Overloaded nodes can appear failed.
- Packet loss can trigger many indirect probes.
- Very large groups need careful dissemination budgets.
The protocol scales the core mechanism; it does not remove every operational constraint.
Failure detection is not failure handling
SWIM tells a member that another process is suspected or failed. It does not decide what application state should move, who becomes leader or whether a quorum remains safe.
Those decisions belong to higher layers:
SWIM observation → routing removes endpointSWIM observation → storage begins replica repairSWIM observation → scheduler replaces workerFor correctness-critical ownership, a consensus or lease mechanism may still be required. Two sides of a partition can both believe the other side failed.
A Go-shaped model
A simplified member state might be:
type Status uint8 const ( Alive Status = iota Suspect Failed) type Member struct { Address string Status Status Incarnation uint64}Updates should be compared by incarnation and state precedence, not arrival time alone. The network can duplicate and reorder packets.
The probe loop, membership table and dissemination queue also need single ownership or synchronization. A protocol description can be race-free on paper while its implementation corrupts local state.
What I keep from the paper
SWIM gives me three durable ideas:
- Separate detecting a change from spreading it.
- Use indirect observers before acting on one failed path.
- Represent uncertainty explicitly with suspicion.
When I see a gossip-based system, I now ask what is actually gossiped, how updates are versioned, how quickly they expire, how missed updates are repaired and which decisions are safe under temporarily different views.
Gossip is a dissemination strategy. Its usefulness depends on the state machine carried inside it.