Coordination & Transactions

Why this module exists

Most distributed failures are not caused by syntax bugs; they are caused by partial success:

• one service commits while another times out,
• a retry runs after ownership changed,
• compensation runs twice and over-corrects state.

This module builds a practical mental model for coordinating multi-step writes when networks are unreliable and processes crash.

Failure model we will use

Assume all of these can happen at any time:

• messages can be delayed, duplicated, or dropped,
• nodes can crash and restart,
• timeouts are not proof of failure,
• clients may retry requests.

Design implication: correctness must come from protocol state, not from timing assumptions.

Running scenario

Use this scenario across the whole module:

• Step 1: reserve inventory
• Step 2: charge payment
• Step 3: create shipment

What can go wrong:

• payment succeeded but shipment step failed,
• coordinator decided commit but crashed before notifying all participants,
• stale leader continues writing after a lease handoff.

Our three tools for this:

• Saga compensation for business rollback,
• 2PC for atomic all-or-nothing decision,
• fencing tokens for stale writer protection.

Two-Phase Commit (2PC): exact contract

2PC has one coordinator and multiple participants.

Phase 1: prepare

Coordinator asks each participant to prepare. Each participant votes:

• commit: "I can commit and I promise to obey final decision"
• abort: "I cannot safely commit"
• timeout / missing / malformed vote: treated as non-commit

Phase 2: decision

• commit only if every participant voted commit
• otherwise abort

Once decision is chosen, coordinator must keep retrying delivery to any participant that has not acknowledged the decision.

Important nuance: blocking behavior

If coordinator crashes after participants prepared but before they learn final decision, participants may block waiting. This is why 2PC is simple and useful, but not always ideal under long coordinator failures.

2PC crash points and consequences

1. Crash before collecting all votes: Outcome can still be abort safely.
2. Crash after deciding but before all acks: Must recover durable decision and retry notifications.
3. Duplicate vote messages: Use per-participant latest state and deterministic decision rules.

Design rule: make decision function pure and deterministic from coordinator view, then make delivery retry idempotent.

Saga: orchestration for long workflows

A saga executes forward steps in order. When a failure is encountered, compensations run in reverse order for already successful steps.

Why saga exists

Some operations are not easy to lock globally for 2PC (external APIs, long human-in-the-loop tasks, non-transactional services). Saga accepts eventual consistency and fixes with compensating actions.

Compensation safety checklist

Every compensation should be:

• idempotent (safe to run more than once),
• scoped (only undo what that step produced),
• ordered (reverse of successful forward effects),
• selective (do not compensate steps that never succeeded).

Retry nuance

Execution logs may contain duplicates or retries. If a success record is replayed multiple times, compensation planning must avoid double undo.

Fencing tokens: stale writer defense

A lock grant carries a monotonically increasing token per resource. Only operations with the current token are allowed to mutate state.

Why this matters

Without fencing, a paused old leader can resume and overwrite newer writes. With fencing:

• grant token 5 to leader A,
• later grant token 6 to leader B,
• any write from A with token 5 is rejected.

Operational nuance

Real systems also need idempotency keys for client retries. If the same write is retried, return the same decision instead of applying twice.

Choosing between 2PC, Saga, and fencing

Use 2PC when:

• participants support prepare/commit protocol,
• strict atomicity is required,
• coordinator availability is acceptable.

Use saga when:

• workflow spans long-running/external steps,
• compensations are available and trustworthy,
• temporary inconsistency is acceptable.

Use fencing whenever:

• ownership can change over time,
• stale actors may continue sending writes,
• lease or lock expiration is part of design.

These are complementary, not mutually exclusive. A common production pattern is: saga for business process + fencing at write boundary + idempotency for retries.

Common implementation mistakes

• deciding commit on partial votes,
• treating timeout as neutral instead of abort,
• compensating failed steps (instead of successful prior steps),
• accepting stale-token writes after leadership changes,
• reapplying duplicate operation IDs.

If your implementation does any of these, it is unsafe under retries/crashes.

What you will build (progressive sequence)

1. Saga Compensation Planner Build rollback order from an execution log with retries and selective compensability.
2. 2PC Decision + Recovery Plan Compute deterministic final decision and identify participants that still need decision replay after crash/restart.
3. Fencing Tokens + Idempotent Writes (capstone) Simulate lock grants and writes; reject stale tokens and return stable results for duplicate operation IDs.

By the end, you should be able to reason about correctness under partial failure, not just happy-path protocol diagrams.