System Design Foundations: Turn a Request into a Testable Contract

Before choosing infrastructure, produce a one-page requirements and capacity argument. This module's deliverable is a design artifact, not a coding problem. You need ratios, unit conversions and the distinction between a serial and a parallel operation. Carry this artifact into Core Building Blocks, Reliability and the Architecture Capstone.

Define success at an observable boundary

Work on a hypothetical notification inbox. One million active users create 12 notifications per day each. Each stored notification occupies 500 bytes before indexes and replication, and retention is 30 days. These are explicit exercise assumptions, not measured production facts.

Define operations separately:

Operation	Example contract	Evidence needed
Create	A successful response means the record survives loss of one storage node	Acknowledgment point and failure test
List inbox	99% of successful reads finish within 200 ms over a rolling 7-day window, at the stated load	Client-observed latency distribution
Read after own create	The user's next authoritative inbox read contains the acknowledged record	A two-operation history, including replica lag
Search old notifications	Results may lag committed data by up to 30 seconds	End-to-end indexing delay, not just worker CPU

Add an availability target, such as 99.9% successful eligible requests over the same 7 days. Define eligibility, success, timeouts and invalid-request treatment; otherwise the ratio can be improved merely by excluding failures. Availability, latency, freshness and durability are distinct. A fast stale response may meet latency while violating freshness; a successful acknowledgment followed by data loss violates durability.

Google's SLO guidance describes turning user needs into indicators and explicit objectives. The values above are teaching assumptions to negotiate with a product owner, not universal SLOs.

Estimate with units and a sensitivity range

Average writes are 1,000,000*12 / 86,400 ≈ 139/s. Assume peak traffic is 8 times average: about 1,112 writes/s, rounded up for planning. A single “QPS” number hides the read/write mix, fanout and payload size; estimate each operation.

Raw daily bytes are 12,000,000*500 = 6,000,000,000, or 6 GB using decimal units. Thirty days is 180 GB raw. If indexes add 50% and there are three copies, plan 180*1.5*3 = 810 GB, before backups, transient compaction space and spare capacity. Replication multiplies stored copies, not the count of user events.

Now challenge the assumptions: if payload size doubles and retention becomes 60 days, raw storage becomes 720 GB and the same factors yield 3.24 TB. A range of plausible answers is more informative than false precision in a single estimate. A 500-byte average also says nothing about the maximum request size or skew from the busiest tenant.

For 2,000 requests/s and mean in-service time 40 ms, average in-service concurrency is about 2,000*0.040=80. This is a steady-work estimate, not the number of workers that guarantees a tail-latency target: queueing, bursts, blocking and resource saturation still need measurement. One worker occupied for 20 ms per operation can finish at most 50/s under the simplified serial model; reaching 100% utilization leaves no spare service capacity for a burst.

Budget a path without adding unrelated percentiles

For one request, serial durations add; simultaneous parallel branches contribute approximately their maximum plus coordination overhead. Suppose an inbox read has 15 ms ingress, 10 ms auth, parallel profile/inbox calls taking 25 and 70 ms, and 5 ms serialization. Its modeled duration is 15+10+max(25,70)+5=100 ms. If the downstream calls were serial it would be 125 ms.

Those are one-request durations or planning allocations, not measured p99 values. Adding each component's p99 does not generally produce end-to-end p99: the slowest events may occur on different requests and can be correlated. Measure the complete request distribution; use traces to explain which branch and wait contributed. Do not sum per-hop percentile dashboards and label the result a validated user-facing percentile.

To allocate a 200 ms deadline, reserve explicit time for transport, queues, service work and response handling. A 150 ms downstream timeout after 80 ms has already elapsed cannot fit that deadline. A budget is a hypothesis to test at peak load and under a slow dependency, not proof of performance.

State what the acknowledgment survives

“A write succeeded” needs a failure boundary. If a leader responds after keeping data only in memory, losing that process may lose acknowledged data. If it responds after a local durable write but before replication, losing the whole node may still lose it. Replicas in the same failure domain do not establish survival of that domain's failure.

For the inbox, document where the record must be durably accepted before success, which failures the design tolerates, and the latency cost of waiting. Separate “not lost” from “immediately visible on every read”: a durable record can still be absent from a lagging replica or asynchronous search index. When uncertain, state the missing measurement or failure test rather than drawing another box.

Reconstruct the argument

1. Change the workload to 600,000 users, 10 notifications/day, 400 bytes each, 14-day retention, 50% index overhead and 3 copies. Compute average writes/s, raw bytes/day and total provisioned data under these factors. Use decimal GB and list omitted overhead separately.
2. A read spends 12 ms at ingress, 8 ms in auth, then 30 and 90 ms in parallel, then 10 ms encoding. Finish its duration. Can two component p99s alone prove this request-path p99?
3. Write one falsification test for “read your own acknowledged write” when reads normally go to a follower 5 seconds behind.

Check the calculations and evidence

1. About 69.44 writes/s; 2.4 GB/day; 2.4*14*1.5*3=151.2 GB. Backups, temporary working space and headroom remain additional, explicit estimates.
2. 12+8+max(30,90)+10=120 ms. No: end-to-end percentiles require the joint request-path distribution, including queues and correlations.
3. Acknowledge a uniquely identified write, immediately issue the session's read while the follower is behind, and verify that the response includes that version or waits/routes according to the contract. A stale success falsifies the promise.

Submit an operation table, three capacity calculations with units, one traced critical path and an acknowledgment/failure statement. The self-review passes only if another reader can recompute the numbers and describe an experiment that could refute each important promise. Later change the workload's peak factor and freshness requirement independently; identify which decisions change rather than redrawing the entire architecture.