Descriptive Statistics

A summary is useful only when you can explain its denominator, units, and limits. This module turns a small dataset into center, spread, and a relationship measure using the exact five functions in statistics-from-scratch.

Trace center and spread together

Take data=[2,4,4,6]. Work from observations to deviations before writing loops:

Observation	Deviation from mean 4	Squared deviation
2	-2	4
4	0	0
4	0	0
6	2	4
Total	0	8

The mean is 16/4=4. Sorting gives [2,4,4,6]; the median averages indices n//2-1 and n//2, so it is (4+4)/2=4. For odd n, choose the single central index n//2. Sort a copy: computing a summary should not rearrange the caller's observations or break their pairing with another feature.

variance(data, population=True) divides the squared-deviation sum by n: 8/4=2. Its square root is std(data)=sqrt(2). If the observations are measured in seconds, variance has squared-second units and standard deviation has second units. Standard deviation is a root-mean-square deviation, not the arithmetic mean of absolute deviations.

Choose the denominator for the question

If the four observations are the entire population being described, dividing by 4 summarizes that population. If they are an independent, identically distributed sample from a larger population with finite variance, sample variance divides by n-1: here 8/3.

Why lose one degree of freedom? After estimating the mean from these observations, their deviations sum to zero. Once three deviations are known, the fourth is constrained. The n-1 correction makes sample variance unbiased for the population variance under the stated sampling assumptions. Taking its square root does not make sample standard deviation exactly unbiased.

The exercise accepts nonempty finite data, with at least two observations for sample variance or sample standard deviation. Population variance of [7] is zero; sample variance of [7] is outside the contract. Rejecting missing values, mismatched lists, or empty datasets is a separate API design task, not unspecified behavior to guess in this exercise.

Derive correlation from paired deviations

For x=[1,2,3] and y=[1,3,2], the means are 2 and 2. Keep each pair together:

centered x: [-1, 0, 1]
centered y: [-1, 1, 0]
sum of pair products = 1 + 0 + 0 = 1
sum of x squares = 2
sum of y squares = 2
r = 1 / sqrt(2×2) = 0.5

In general, divide the sum of paired deviation products by the square root of the product of the two sums of squared deviations. The factors of n cancel when covariance and both variances use the same convention. Dividing only the numerator by n incorrectly scales the answer.

Sorting x and y independently destroys the observed pairs. In this example it would change y to [1,2,3] and falsely produce r=1. Lists must have equal length and contain at least one pair. If either list is constant, the mathematical denominator is zero; this exercise returns 0.0 by policy. That sentinel is not evidence that a measurable correlation is zero.

Challenge the summaries with transformations

Add 100 to each observation in [2,4,4,6]. The mean and median increase by 100, but centered deviations and variance stay unchanged. Multiply every observation by -3: variance is multiplied by 9 and standard deviation by 3, the absolute scale factor.

Now replace the last value with 60. The median remains 4, while the mean becomes 17.5. Neither is an error: the mean incorporates the size of every observation; the median depends on central order. Investigate whether an extreme observation is a measurement error or a meaningful event before removing it.

Zero Pearson correlation does not imply independence. For x=[-1,0,1], let y=x²=[1,0,1]. The positive and negative paired deviations cancel, giving zero linear correlation even though y is completely determined by x. A scatterplot reveals a relationship that the single coefficient hides. Correlation alone also does not identify a causal direction or rule out a common cause.

Finish a variance trace

For [1,3,5], the mean is 3 and squared deviations are [4,0,4]. Find population variance, sample variance, and the median without running the reference. Then predict what multiplying the data by 2 does to each standard deviation.

Reveal the calculation

Population variance is 8/3, sample variance is 8/2=4, and the median is 3. Both standard deviations double; both variances quadruple. Adding a constant afterwards changes neither variance.

Connect descriptive summaries to modeling

A z-score (x-mean)/std expresses distance in units of standard deviation; it does not make a skewed distribution Gaussian. Fit those statistics on training data and reuse them on held-out data, as the feature-engineering module will require.

The familiar 68/95/99.7 percentages describe a normal distribution, not every dataset. Under the usual independent, identically distributed, finite positive variance assumptions, the central limit theorem concerns the distribution of the standardized sample mean as sample size grows. It does not say the original observations become normal, and dependent or heavy-tailed samples need separate care. See MIT's probability text, central limit theorem.

Implement mean and variance first, reuse variance in standard deviation, then implement median and paired correlation. Independently check shift/scale behavior, reversed order of complete pairs, constant columns, and input preservation. These checks test meaning as well as familiar numeric examples.