System design: decisions need evidence

Worked example: a regional chronological news feed.

Deliver requirements, capacity arithmetic, data ownership, request/failure traces, a rejected alternative, and measurable acceptance criteria.

All workload and service-rate figures are course assumptions to test.

Workload and targets

• 10 million viewers × 20 reads/day = 200 million/day ≈ 2,315 reads/s average.
• Plan for 25,000 reads/s and 250 ordinary posts/s peak.
• Mean 200 recipients/post → 50,000 feed-reference inserts/s.
• Regional feed p99 target: 200 ms; steady-state freshness: 99% within 5 s.
• Seven-day public immutable posts; global failure handling needs an extension.

Why hybrid fanout?

Eight assumed 10,000-insert/s workers provide 80,000/s total.

One 10-million-follower post takes:

• 125 s with all capacity and no other work.
• About 333 s of spare capacity while ordinary work consumes 50,000/s.

Merge exceptional-author posts at read time; measure the extra read cost.

Make records and keys explicit

• Posts + request result + outbox commit atomically before acknowledgement.
• A retry of the same create request returns the same post ID.
• The outbox relay can publish twice.
• ViewerFeed has unique (viewer_id,post_id) references.
• A retrying worker upserts; post bodies remain authoritative in Posts.

Replication configuration must support the promised single-zone durability.

Bound the read

• Page size 50; at most 20 followed exceptional authors.
• Fetch 50 ordinary references + 50 from each exceptional stream.
• At most 1,050 candidates/read; merge, deduplicate and hydrate 50 bodies.
• At 25,000 reads/s: up to 26.25 million candidates/s before reuse.
• A (created_at,post_id) cursor handles ties; async fanout still permits misses during a pagination session. This is not a snapshot guarantee.

Trace the failure

1. Create commits, response disappears → retry returns the persisted post ID.
2. Relay publishes, crashes → duplicate publication is permitted.
3. Worker writes, crashes before acknowledgement → same recipient/post upsert.
4. Workers stop 120 s at 50,000/s → 6 million queued reference writes.
5. Recovery service 80,000/s minus arrivals 50,000/s → 30,000/s spare.
6. Drain takes 200 s after restart, assuming storage sustains the rate.

Validate the claims

• Measure end-to-end latency; summing stage p99s is not a proof.
• Measure warm/cold caches and skewed follower counts.
• Inject crashes at acknowledgement boundaries.
• Verify surviving acknowledged posts after zone loss.
• Measure pending age and recovery time, not only queue length.

A spreadsheet estimate is a hypothesis until measured.

Transfer and assess

• Dispatch: location throughput and atomic ride/driver reservations.
• Video: byte traffic, cache misses and cold-origin capacity.
• Analytics: overlapping replay, output identity and spare recovery capacity.

Artifact rubric: requirements, capacity, ownership, request path, recovery, trade-off, validation. Aim for 11/14 with complete failure-recovery evidence. The quiz checks reasoning; it does not grade the freeform document.