PCA: A Projection You Can Explain

Training rows: [[1,2],[3,4],[5,6]].

Can one coordinate preserve all their variation?

Prerequisites: dot products, means, covariance.

1 / 8

Fit the coordinate system

mean = [3,4]
centered = [[-2,-2],[0,0],[2,2]]
C = [[8/3,8/3],[8/3,8/3]]

The exercise divides by n, not n-1.

A unit direction v captures variance v.T @ C @ v.

2 / 8

Predict the projection

v = [1/sqrt(2),1/sqrt(2)]
lambda = 16/3
Z = [[-2*sqrt(2)],[0],[2*sqrt(2)]]

Flip v's sign. What changes?

Coordinates flip; reconstruction and variance do not.

3 / 8

Power iteration amplifies a direction

For C=diag(4,1), direction ratios evolve:

[1,1] -> [4,1] -> [16,1]

Normalize after each multiplication.

Convergence needs an overlapping start and an eigenvalue gap.

4 / 8

Deflate, then guard against roundoff

C_remaining = C - lambda*v*v.T

Before normalizing a later candidate, subtract its projections onto previous components.

A second copy of the first component is not a second signal.

Rank exhausted? Complete with unit orthogonal directions of zero training variance.

5 / 8

Keep the training frame

components: (k,d), directions stored as rows
Z = (X-mean) @ components.T
X_approx = Z @ components + mean

New row [7,8] uses training mean [3,4].

Its coordinate is 4*sqrt(2), not zero.

6 / 8

Variance is a choice, not a label guarantee

Centering removes location; scaling changes the objective.

Fit preprocessing only on training data.

With eigenvalues [6,3,1], two components keep 90% of variance.

Can the discarded 10% contain class information? Yes: PCA never saw labels.

7 / 8

Evidence that your implementation works

  • Numeric covariance agrees with the declared denominator.
  • Components have unit length and pairwise zero dot products.
  • Rank-one data does not produce duplicate components.
  • Transform uses the saved training mean.
  • Reconstruction works up to numerical tolerance and sign choices.
8 / 8
←/→ or click edges to navigate · ? help · N notes · F fullscreen