Observability & Resilience

Follow one gateway during a slow dependency incident: measure user-visible failures, limit admission, bound waiting work, schedule retries, then control recovery probes. These exercises are deterministic models; a passing trace is not a load test. Start with fractions, ordered events, and saturating arithmetic.

Turn request counts into an SLO signal

SLOMetrics considers latency at or below threshold good. With target 0.99, the allowed bad fraction is 0.01; burn is (bad/total)/0.01. A burn of 4 means four times the allowed fraction, not four percent of a time budget already used.

Use threshold=200 and minute buckets [100,100,100,300], [100], [400,100]:

Window	Requests	Bad	Burn
all three minutes	7	2	28.571...
newest minute	2	1	50
newest two minutes	3	1	33.333...

Do not average per-minute percentages: the middle minute has one request and the first has four. Weight by requests. Sorted latencies are five 100s, 300, 400, so nearest-rank p99 is 400: ceil(0.99*7)-1=6. A small sample's p99 may just be its maximum; it does not estimate a stable population tail by itself.

At fast threshold 40 and slow threshold 30, both windows breach, so Alert=true. Change the slow threshold to 40 and Alert=false even though the fast window still breaches. Empty total history returns no alert. The model records latency only; a real SLI must specify what happens to errors, rejected requests and missing measurements. See the SRE workbook's alerting model.

Metrics indicate scope, traces locate a slow branch, and logs explain individual admission/probe decisions. A low CPU average does not rule out a blocked request path or a saturated dependency.

Admission and queueing use different state

A token bucket bounds admission bursts. With capacity=4 and refillRate=2:

Request (time,cost)	Tokens after refill	Allowed	Tokens after
(0,3)	4	yes	1
(0,2)	1	no	1
(1,2)	3	yes	1
(1,1)	1	yes	0
(3,4)	4, capped	yes	0

Rejected requests do not spend tokens. Same-time requests do not refill between them. Normalize an out-of-order time to the previous effective time; never move the refill clock backward. Multiply elapsed by rate in int64 before clamping.

Admitted requests may still wait. BackpressureSim orders each tick as arrive, shed, record peak, serve. At capacity=5, initialQueue=2:

Tick (arrivals,service)	Before shedding	Drop	Peak candidate	End queue
(5,4)	7	2	5	1
(2,1)	3	0	3	2

TotalProcessed=5, TotalDropped=2, MaxQueue=5, EndQueue=2. Conservation: initial 2 + arrivals 7 = processed 5 + dropped 2 + remaining 2. Serving before shedding would admit work this model rejects. Recording only post-service queue would hide the peak of 5. A rate limiter cannot replace a queue bound when service slows below the admitted rate.

Reconstruct retries without inventing guarantees

RetrySchedule(start=10, outcomes=[false,false,false,true], baseDelay=3, maxDelay=5) emits [10,13,18,23]. Delays are 3,5,5; the first entry is the initial attempt. Success stops the sequence; later supplied outcomes are ignored. Addition and doubling saturate at MaxInt. With cap zero, retries have equal timestamps: this deterministic boundary is not a recommended production policy.

The exercise has no operation durations, deadlines or jitter. For an independent extension, give every attempt a timeout of 4 and the whole operation a deadline of 20, starting at 0. Attempt 0 times out at 4; wait 3, attempt 1 times out at 11; wait 5, attempt 2 starts at 16 and can run only to 20. There is no budget for a fourth attempt. State whether a deadline-truncated attempt is permitted before implementing a scheduler with durations.

Three attempts at each of three independently retrying layers can produce 3*3*3=27 downstream attempts. Cap a shared retry budget, add jitter, and use idempotency for effects. Backoff alone does not prevent duplicate payment.

Work the circuit-breaker boundary

With failureThreshold=2 and resetTimeout=5:

Event	Allowed	Probe	State after	Explanation
failure at 0	yes	no	closed	failure count 1
failure at 1	yes	no	open	triggering request already ran
success at 3	no	no	open	outcome ignored; no call made
failure at 6	yes	yes	open	exact boundary; failed probe restarts timer
success at 7	no	no	open	new cooldown runs from 6
success at 11	yes	yes	closed	successful probe resets failures

Half-open is transient in this API: Probe=true identifies it; State reports only closed/open after evaluation. Rejecting at the exact timeout boundary or counting a rejected event's Success field changes the protocol. Production concurrent callers need an atomic probe permit; this sequential simulator has no overlapping probes to coordinate.

Connect and challenge the controls

Suppose service capacity drops from 100 to 40 requests per tick while the limiter continues admitting 80. A bounded queue eventually sheds at least 40 per tick in steady state unless admission or offered load falls. Increasing queue capacity delays that point; it does not create processing capacity. Retries that re-enter the same queue can worsen the deficit.

Predict the final two breaker rows if the probe at 6 succeeds instead.

Check the continuation

The breaker closes at 6. Both later successes are normal allowed events with Probe=false and State=closed. Keeping the old openedAt value is harmless only if closed-state evaluation never consults it.

Implement the five models, then write a fresh incident trace with a recovering dependency. Explain which signal should change admission and which request may probe. Revisit later with two callers at the cooldown boundary and identify what synchronization the sequential API leaves for the integration layer.