Reliability: bound the work when a dependency fails

A useful reliability design states what users experience, what resources remain bounded, and how service resumes. Naming timeouts and retries is insufficient: their interaction can multiply load precisely when a dependency has least capacity. The following numbers are teaching assumptions, not universal tuning recommendations.

Start with a measurable user outcome

Suppose a checkout request has a 600 ms end-to-end deadline. Define a good event as a correct checkout result delivered within that deadline, and target 99.9% good events over the chosen reporting window. For 2,000,000 eligible requests, the error budget is 2,000 bad events. A fast incorrect charge is bad; a correct response after the deadline is also bad. Decide explicitly whether deliberate admission rejections count against the user-facing SLI; silently excluding them can make an unavailable service look healthy.

Record traffic, errors, latency distributions and saturation. Separate first attempts from retry attempts, include queue wait in end-to-end latency, and trace the same request across dependencies. An average latency of 40 ms can hide a small population of multi-second requests. A 99th-percentile chart alone does not show how many requests failed to reach the service.

Derive a deadline and retry schedule

Reserve 50 ms for authentication and local processing and 50 ms to assemble and return the response. That leaves 500 ms for the payment dependency. Permit at most three attempts, each capped at 140 ms. After failures, sample full-jitter backoff from [0,30] ms and [0,50] ms. The worst-case budget is 3*140 + 30 + 50 = 500 ms.

A worst-case timeline is attempt 1 at t=50..190, wait until 220, attempt 2 at 220..360, wait until 410, and attempt 3 at 410..550; final processing ends at 600. Before each attempt, compute remaining deadline and reserve the final 50 ms. Shorten or skip an attempt if the remaining budget cannot support it. The timeout must include the phases the client really waits for, such as connection establishment; a read-only socket timeout can leave gaps.

Use a monotonic elapsed-time source. Cancellation propagates to work that can stop, but cannot undo a remote write that already committed. A timeout is an unknown outcome, not evidence that the server did nothing.

If three layers independently allow three attempts, one user request can cause 3³=27 leaf calls. Choose one layer to own retries, bound attempts and total elapsed time, and apply a retry budget shared across requests. For example, 1,000 initial calls/s with at most 100 retry calls/s caps admitted downstream attempts at 1,100/s. Jitter spreads wakeups; it does not itself cap total work. Retry transient failures when safe; malformed input and failed authorization will not improve by repeating them.

Trace an ambiguous payment outcome

At t=170, the provider durably records payment P. Its response is lost; at t=190 the client times out. Retrying with a new operation ID may charge twice. Instead use the same idempotency key, scoped to the tenant and operation, with a request fingerprint. The service records the key, request and result consistently with the side effect. Duplicate identical requests return the prior result; reuse of the same key with a different amount is a conflict.

Two concurrent requests must not both observe an absent key and perform the charge. A unique claim or equivalent atomic transaction coordinates them. A process crash between an external charge and a separate local key write still leaves a gap; forward the key to an idempotent provider or reconcile the unknown outcome. Retaining keys for 24 hours does not protect a duplicate arriving after 24 hours. Choose retention from the actual retry/replay horizon. An idempotency key is not an authorization credential.

Bound overload before adding retries

Assume admitted work arrives at 120 requests/s and workers complete 100/s. A queue of capacity 80 fills from empty in 80/(120-100)=4 seconds under this fluid approximation. At full depth it represents about 80/100=0.8 seconds of service, already longer than our 600 ms deadline. Increasing the queue merely postpones rejection and can increase useless work.

Use bounded in-flight calls, queue limits and deadline-aware admission. Isolate payment calls from optional recommendation calls with separate concurrency limits, so a recommendation outage does not consume every worker. Shed optional recommendations first; checkout may return a plain confirmation. If payment capacity is exhausted, fail quickly or report an explicitly pending status according to the product contract; never invent successful payment.

A circuit breaker is an admission state machine. In this example, three consecutive transport failures open it for 10 seconds. During that interval requests fail fast. Afterward one half-open probe is admitted: success closes the breaker, failure opens it for another interval. Limit probe concurrency so all waiting callers do not become probes. These thresholds are exercise policy, not recommendations that fit every workload.

Practice with checkable answers

1. The response is lost after a charge. What is safe to retry? Answer: the same scoped idempotency key and request, with provider-side deduplication or reconciliation; a new key creates a new operation.
2. Queue capacity is 90, arrival rate 150/s, service rate 120/s. Answer: it fills in 3 seconds from empty; its fluid drain time is 0.75 seconds with no new arrivals. A larger queue does not repair sustained overload.
3. Two layers each allow four total attempts. Answer: up to 16 leaf calls; place ownership and a shared budget explicitly.
4. The half-open probe fails. Answer: reopen and admit no ordinary calls until the next allowed probe interval.

AWS describes the operational rationale for timeouts and bounded retries and idempotent API contracts. Google's SRE chapter on handling overload discusses admission and degraded service. The schedules and arithmetic above are independently constructed exercises; no simulator or production integration is implied by this design-only module.