Membership & Failure Detection

Why this module exists

In distributed systems, node liveness is never perfectly known. You can only infer health from delayed signals (heartbeats, probes, gossip).

If membership is wrong, everything above it breaks:

• traffic routes to dead nodes,
• leaders keep stale peers in quorum sets,
• rebalancers move data based on outdated topology.

This module focuses on practical failure-detection correctness under noisy networks.

Failure model and reality check

Assume:

• packet delay and loss are common,
• processes pause (GC, CPU starvation) without crashing,
• clocks are imperfect,
• temporary partitions occur.

Consequence: a node can look dead to one observer and alive to another. Design goal is not certainty; it is fast convergence with bounded false positives.

Membership gossip: what actually propagates

Nodes exchange digests describing latest known version of each member. A digest entry typically has:

• member identity,
• incarnation/generation,
• heartbeat counter.

Merge logic is directional:

• if local entry is newer, send update,
• if remote is newer, request update,
• if equal, no action.

Nuance: incarnation dominates heartbeat. A node restart should beat old heartbeat counters from before restart.

Digest merge pitfalls

Common bugs:

• comparing only heartbeat without incarnation,
• failing to deduplicate duplicate entries,
• non-deterministic merge output ordering,
• treating unknown members as errors instead of sync deltas.

Membership propagation must be deterministic; otherwise gossip rounds oscillate.

Phi accrual detector intuition

Phi accrual turns "time since last heartbeat" into a suspicion score.

Approximation used in this module: phi = (elapsed / mean_interval) / ln(10)

Where:

• elapsed is now minus last heartbeat timestamp,
• mean_interval is average heartbeat gap from recent samples.

phi grows smoothly as heartbeats are delayed. You choose a threshold (for example 5, 8, 12) based on desired sensitivity.

Phi operational nuance

• Too few samples: suspicion is unstable.
• Low threshold: faster failover, more false positives.
• High threshold: fewer false positives, slower failover.
• Clock jumps and negative elapsed need explicit handling.

Rule of thumb: suspicion should trigger workflow transitions, not immediate hard delete.

SWIM suspicion lifecycle

SWIM-style lifecycle is usually: alive -> suspect -> dead

Key controls:

• unique reporters to avoid one noisy node dominating suspicion,
• suspicion threshold/quorum before transitioning to suspect,
• alive reports with newer incarnation to clear stale suspicion,
• optional confirm step before marking dead.

This balances safety (avoid false death) and liveness (eventual cleanup).

Incarnation numbers matter

If node A is suspected and then restarts, it should advertise higher incarnation. Peers must treat higher-incarnation alive as fresher truth.

Without incarnation handling:

• stale suspect reports can keep a healthy restarted node quarantined.

Tuning tradeoffs

You tune three coupled dimensions:

• gossip dissemination speed,
• detector sensitivity (phi threshold / probe timings),
• suspicion quorum for state transitions.

Aggressive tuning recovers faster but can destabilize under transient latency. Conservative tuning is stable but slow during real failures.

Common implementation mistakes

• using raw heartbeat count without generation/incarnation,
• counting duplicate suspect reports from same reporter multiple times,
• forgetting to reset suspicion state on alive with newer incarnation,
• marking dead directly from one weak signal,
• mutating output order between runs.

What you will build (progressive sequence)

1. Versioned gossip digest sync planner Compute deterministic send/request sets with generation+heartbeat ordering.
2. Phi accrual evaluator Derive mean heartbeat interval from samples and produce suspicion decision.
3. SWIM round transition engine (capstone) Apply ordered reports with reporter dedupe, incarnation handling, and suspect/dead transitions.

By the end, you should be able to reason about membership convergence and failure suspicion as protocol state machines, not as ad-hoc timeouts.