Self-Attention: Compute the Mixture, Then Prove It Is Causal

Give rows and columns meanings

Q, K, V: (T,D). Scores: (T,T).

Score row i belongs to destination i; column j belongs to source j.

Compare Q[i] with K[j], then mix V[j].

Scale the comparison

score[i,j] = dot(Q[i],K[j])/sqrt(D).

Scaling uses feature width. Under independent unit-variance components, unscaled dot-product variance grows with D.

Predict the visible mixture

At position 1: scores [0,log(3),99].

Values [2,4], [6,0], [100,100].

Which score is excluded? What are the remaining normalized weights?

Resolve the numeric trace

Masked scores [0,log(3),-inf].

Weights [1/4,3/4,0].

Mixture [5,1]. Future score and future value contribute nothing.

Mask before normalization

Each row permits j<=i, including its diagonal.

Use negative infinity above the diagonal; finite sentinels can be overcome by larger scores.

An all-masked row is outside this softmax contract.

Find the wrong axis

Each destination needs its own probability distribution over sources.

Normalize rows. Global normalization makes destinations compete.

Zeroing weights after softmax leaves the denominator contaminated by future scores.

Test a counterfactual prefix

Change only future inputs. Earlier outputs must match in causal mode.

An earlier output must also have zero gradient with respect to future inputs.

This tests the entire forward path, not just the mask helper.

Reconstruct the pipeline

Full projections -> scaled scores -> mask -> row softmax -> weighted values -> output projection.

Core attention: O(T*T*D) work, O(T*T) scores.

Dense projections additionally cost O(T*D*D).