Observability & Resilience

Why this module exists

Distributed systems usually fail gradually, not instantly:

• tail latency increases before hard errors appear,
• retries increase traffic and hide root cause,
• one overloaded dependency starts a cascade.

Observability tells you what users are experiencing. Resilience controls decide how the system behaves while degraded. You need both or you will either fail blindly or react too late.

Failure model for this module

Assume all of these can happen at any time:

• dependencies become slow without fully failing,
• requests arrive in bursts,
• clients retry aggressively,
• some instances recover while others are still overloaded.

Design implication: optimize control loops for degraded conditions, not just steady-state throughput.

Running scenario

Use this scenario across all exercises:

• API gateway receives user traffic.
• Gateway fans out to Auth, Catalog, and Payment.
• A partial regional incident makes Payment slow and intermittent.

Questions we must answer:

• How quickly can we detect user-visible impact?
• How do we prevent retries from overwhelming the recovering service?
• Which requests should be admitted, queued, delayed, or rejected?

Observability model: signal types and purpose

Treat observability as a layered model, not one metric dashboard:

1. Metrics: fast aggregate health (latency, error rate, queue depth).
2. Logs: discrete event history and decision traces.
3. Traces: end-to-end causality for fan-out requests.

Common production mistake:

• alert on resource metrics only (CPU, memory) and miss user impact.

Better practice:

• alert from user-facing SLO signals first,
• use logs/traces for diagnosis after alert triggers.

SLO engineering: from target to action

An SLO is not just a percentage; it is an operating policy.

Core terms:

• SLI: measured user-facing indicator (for example, request latency <= 250ms).
• SLO target: fraction of requests that must satisfy SLI (for example, 99.9%).
• Error budget: allowed bad fraction (1 - target).
• Burn rate: how fast current behavior consumes budget.

Why burn rate matters:

• Error rate alone does not encode urgency.
• Burn rate directly answers: "How long until budget is exhausted?"

Operational nuance:

• One short spike can be noisy.
• One long mild degradation can still bankrupt the budget.
• Use fast and slow windows together to avoid both false positives and slow detection.

Tail latency and fan-out risk

User experience is usually constrained by tail behavior, not averages.

If one request fans out to many internal calls, end-to-end p99 worsens because the slowest branch dominates.

Practical implications:

• Track p95/p99/p999, not just mean.
• Prefer histograms/quantiles over only rolling averages.
• Separate "server busy" from "dependency slow" in instrumentation.

Overload control: admission before collapse

When demand exceeds service capacity, queues and retries can create positive feedback loops. You need explicit admission control.

Key levers:

• Token bucket rate limiting: smooth burst admission at ingress.
• Backpressure + bounded queues: keep core services stable and shed excess.
• Load shedding policy: reject early when queued work is unlikely to complete on time.

Design rule:

• bounded queues are a safety mechanism, not a failure.
• unbounded queues are often hidden outages.

Retry discipline

Retries can improve success rate or trigger collapse.

Safe retry policy requires:

• request deadlines/timeouts,
• exponential backoff with cap,
• jitter to avoid synchronized retry waves,
• idempotent operations for correctness under repeats.

Anti-pattern:

• "retry immediately until success" under dependency saturation.

Circuit breakers as failure containments

Circuit breakers separate normal traffic from recovery probing:

• Closed: normal traffic; watch consecutive failures.
• Open: reject quickly to reduce pressure.
• Half-open: allow controlled probe to test recovery.

Tuning tradeoff:

• trip too late -> cascade spreads,
• trip too early -> availability drops unnecessarily.

The right threshold and reset timeout depend on failure mode and traffic shape.

Control-loop interactions (most important nuance)

These controls do not operate independently:

• retries increase request volume,
• larger queues increase latency and timeout probability,
• more timeouts trigger more retries,
• breakers and rate limiters can stabilize the loop if tuned coherently.

Think in feedback loops:

• detect -> decide -> act -> measure again.

If your loop does not reduce pressure under stress, it is not resilience.

What you will build (progressive sequence)

1. SLO Tail-Latency Analyzer Compute p99 and multi-window burn rates to turn raw latency into actionable alerts.
2. Token Bucket Admission Controller Simulate deterministic ingress admission under bursty traffic.
3. Backpressure Queue Simulator Model bounded queue behavior, shedding, and throughput under sustained overload.
4. Retry Backoff Scheduler Build deterministic retry timelines with capped exponential delays.
5. Circuit Breaker Controller (capstone) Simulate closed/open/half-open behavior across realistic failure-recovery traces.

By the end, you should be able to explain not only each control in isolation, but also how to tune them together to keep user impact bounded during incidents.