State the user-visible contract

600 ms checkout deadline; correct result required. 99.9% good events over 2,000,000 requests → 2,000-event budget.

A fast incorrect charge and a late correct result both fail this SLI. Decide how admission rejections are counted.

1 / 6

Spend a finite deadline

50 ms local + 500 ms dependency + 50 ms return.

Three 140 ms attempts + backoffs at most 30 and 50 ms = 500 ms. Attempt starts in worst case: 50, 220, 410 ms.

Recompute remaining time; connection setup and queueing consume time too.

2 / 6

Count the amplification

Three layers × three attempts per layer → up to 27 leaf calls.

One retry owner and a shared budget bound work. 1,000 initial/s + at most 100 retries/s → at most 1,100 attempts/s. Jitter changes timing; it does not enforce that bound.

3 / 6

Timeout means unknown outcome

Payment committed at 170 ms; response lost; timeout at 190 ms.

Retry the same scoped key and fingerprint. Atomic deduplication must cover the side effect; an external provider needs its own idempotency or reconciliation contract.

4 / 6

Predict queue failure

Arrivals 120/s; service 100/s; capacity 80.

Queue fills in 4 seconds. Full queue holds about 800 ms of work. Increasing capacity cannot fix sustained overload. Isolate optional work and reject requests that cannot meet their deadline.

5 / 6

Breaker recovery needs a bound

Three failures → open for 10 seconds → one half-open probe. Success closes; failure reopens.

Explain why admitting every waiting caller as a probe recreates the outage. Thresholds here are teaching policy, not production defaults.

6 / 6
←/→ or click edges to navigate · ? help · N notes · F fullscreen