Observability & Resilience

1 / 10

Core challenge

Distributed incidents are usually feedback loops:

  • slow dependency -> timeouts
  • timeouts -> retries
  • retries -> more load

Goal: detect early and reduce pressure before collapse.

2 / 10

Observability model

  • Metrics: fast health and SLO tracking
  • Logs: decision history and event detail
  • Traces: cross-service causality

Alert on user-facing SLO symptoms first.

3 / 10

SLOs and burn rate

  • SLI defines "good request"
  • Error budget = 1 - target
  • Burn rate = errorRate / errorBudget
  • Fast + slow windows catch both spikes and drifts
4 / 10

Tail latency reality

  • Mean latency can look fine while users suffer
  • p99/p999 drive perceived reliability
  • Fan-out amplifies tail risk

Instrument and reason about percentiles, not averages only.

5 / 10

Admission and overload control

  • Token bucket: smooth ingress bursts
  • Backpressure queue: bound in-flight work
  • Load shedding: fail fast when completion is unlikely

Bounded queues protect availability.

6 / 10

Retry policy safety

  • Backoff + cap + jitter
  • Deadline-aware retries
  • Idempotency is required for correctness

Retries without policy can amplify outages.

7 / 10

Circuit breaker behavior

  • Closed: count consecutive failures
  • Open: reject quickly during cooldown
  • Half-open: probe recovery with limited exposure

Tune threshold and reset timeout to avoid oscillation.

8 / 10

Control-loop thinking

  • Controls interact, not isolate
  • Every decision changes future load
  • Validate by simulation and incident traces

Resilience = stable behavior under stress.

9 / 10

Exercise progression

  1. SLO tail-latency analyzer
  2. Token bucket admission controller
  3. Backpressure queue simulator
  4. Retry backoff scheduler
  5. Circuit breaker capstone
10 / 10
Use arrow keys or click edges to navigate. Press H to toggle help, F for fullscreen.