Coordination & Transactions
Work through examples, try the reasoning yourself, then put it into practice.
View SlidesBefore you start
- Trace maps keyed by participant, resource and operation identity.
What you will practice
- Derive decision/retry and compensation plans from their exact deduplication rules.
- Explain which durable fact survives a crash and why replay cannot apply an effect again.
In this lesson
Lesson
6 min readCoordination & Transactions
The shared problem is partial success: inventory is reserved, payment may have succeeded, and shipment is unknown. A timeout does not undo an effect. This sequence separates three responsibilities: choose a transaction decision, plan business compensation, and reject stale ownership. Start with maps, ordered event logs, and the difference between a result and an acknowledgement.
Derive the decision and the delivery plan separately
TwoPhaseCommit consumes participant snapshots, not a network transcript.
The last record for each ID is authoritative. Consider:
| Input record | Vote | Acked | Survives deduplication? |
|---|---|---|---|
| inventory | commit | true | yes |
| payment | timeout | false | no |
| shipment | commit | false | yes |
| payment | commit | false | yes |
The three surviving votes are commit, so Decision=commit. Delivery still owes
[payment,shipment], in lexicographic order. Changing shipment's vote to abort
changes Decision to abort, but the same unacknowledged participants still need
the decision. Acknowledgements control retransmission; they do not count as votes.
Empty participant input aborts by the course rule. A malformed vote also aborts.
Implement deduplication before either decision or retry selection. Otherwise, a stale timeout record can poison a later authoritative commit snapshot, or an old Acked=false can generate a spurious retry after a newer acknowledgement.
A pure planner does not make recovery durable
Real 2PC needs durable prepared state and a durable final decision. Once a commit decision exists, a restart must replay that decision; it must not rerun a vote calculation on an incomplete recovered list and turn commit into abort. A prepared participant that cannot learn the decision may have to wait. The transaction-commit analysis by Gray and Lamport explains this blocking boundary.
The exercise has no transaction ID, persisted decision or disk/network API. It returns a plan for one supplied snapshot. To embed it in a coordinator: first establish the complete participant set and transaction identity; durably record the decision before sending it; recover that decision on restart; retry notification idempotently until acknowledged. These are integration obligations, not guarantees proved by passing the planner tests.
Reconstruct compensation from the successful prefix
SagaCompensations stops at the first non-ok status, including skipped or
malformed statuses. Entries after that boundary cannot retroactively justify
compensation. For the prefix below, eligibility also requires Compensable=true
and a nonempty action.
| Index | Name | Status | Action | Compensable |
|---|---|---|---|---|
| 0 | reserve | ok | release-v1 | yes |
| 1 | charge | ok | refund | yes |
| 2 | reserve | ok | release-v2 | yes |
| 3 | audit | ok | undo-audit | no |
| 4 | ship | failed | cancel | yes |
| 5 | notify | ok | undo-notify | yes |
Boundary=4. Scan eligible successes backward: reserve at 2 contributes
release-v2, charge at 1 contributes refund, reserve at 0 is already seen.
Result is [release-v2,refund]. The latest eligible success wins, not
simply the latest record with the same name: a noncompensable retry does not
erase an earlier compensable success. With no failure boundary, return nil.
This planner orders actions; it does not execute them, persist retries or make compensation succeed. A refund is a new business action and may fail or require human resolution. Give compensations stable operation IDs, scope them to the original effect, and track their outcomes. Reversing a list does not create an atomic rollback across services.
Put fencing at the resource boundary
FencingDecisions stores the greatest granted token per resource and the
first decision per OperationID. New writes require equality with the known
grant. A forged larger token is rejected too. Client is descriptive metadata;
the exercise authenticates neither clients nor grants.
| Event | Stored token for cart | Result |
|---|---|---|
| write op-a with token 4 | absent | reject no-grant, cache rejection |
| grant cart token 4 | 4 | no write decision |
| retry op-a token 4 | 4 | replay original no-grant rejection |
| write op-b token 4 | 4 | accept, cache acceptance |
| grant cart token 5 | 5 | ownership advances |
| retry op-b token 4 | 5 | replay acceptance, no new write |
| write op-c token 4 | 5 | reject stale-token |
| write op-d token 6 | 5 | reject stale-token; 6 is not granted |
Check the idempotency cache before fencing. A replayed accepted decision reports what already happened; it does not authorize applying an old write again. Caching only successes also violates the contract: op-a must keep its first rejection. A new logical attempt needs a new ID. IDs are global across this event stream, so the same ID cannot mean unrelated writes on different resources.
In a real sink, checking the token and applying the effect must form one atomic operation relative to ownership changes. Checking a lease only at the client leaves the paused-client race open. This model has a grant stream visible to the sink; other fencing designs may enforce an increasing accepted-token rule instead of the exact-equality rule used here.
Complete a changed trace
Take the saga prefix above, but make index 2 noncompensable and index 4 skipped. Predict rollback order. Then insert a grant of token 3 after token 5 in the fencing trace and predict whether a fresh token-3 write succeeds.
Check both predictions
Skipped still creates the boundary. The eligible reverse order is refund, release-v1; the noncompensable reserve record cannot hide the earlier eligible one. Grant 3 does not lower the stored maximum 5, so the fresh write is rejected.
Implement saga, 2PC and fencing, then design a crash timeline for the integrated inventory/payment/shipment workflow. Identify the durable fact required at each restart. Your design should say who retries, what key deduplicates the effect, and what remains unknown after a timeout; naming a protocol is not enough.
Practice sequence
Use the order below for a guided path, or choose an item to revisit. A pass records a successful attempt; try a fresh example later to check what you can reconstruct.
- Step 1
Saga Compensation Planner
- Step 2
Two-Phase Commit: Decision and Recovery Plan
- Step 3
Fencing Tokens and Idempotent Writes
- Step 4
Coordination and Transactions Checkpoint
Trace fresh cases, diagnose plausible bugs, and separate the exercise model from production guarantees.