Mini-GPT: Integrate and Verify

Real next-token pairs

Source [0,1,2,0] → inputs [0,1,2], targets [1,2,0].

• Do not invent a repeated final target.
• Each row predicts the immediately following token.

Trace shapes

IDs (T,) → token+position embeddings (T,D) → blocks (T,D) → final norm → logits (T,V).

• Every residual preserves width.
• Linear stores one weight row per output.

Count unique parameters

For V=3,D=4,M=5, one block:

• Embeddings 12+20, block 244, final norm 8, output 12.
• Total 296 unique scalar identities.
• Aliases must not cause double optimizer updates.

Stable loss retains the correction signal

Logits [1000,-1000], target 1:

• Shifted [0,-2000]; log-sum-exp loss ≈2000.
• Gradients ≈[1,-1].
• Clipping an underflowed probability hides this signal.

Prove causality

• Change future tokens; early logits stay unchanged.
• Position embeddings do not excuse a changed first row.
• Exclude future scores before normalization.

Generate and evaluate

• Crop input context, retain full output.
• Sample only the final logit row at positive temperature.
• Verify invariants, fit a tiny case, then evaluate held-out data.
• Loss decrease alone does not establish every contract.