Row t of a (V,V) table contains next-token logits.
Input [0,1,0] reuses row 0 twice.
The table has V squared parameters and remembers only the current token.
Subtract the row maximum before exponentiation.
For [1000,1001], use shifted logits [-1,0].
Keep arithmetic in Value; the common shift changes no probabilities.
Logits [1000,-1000], target 1.
The target probability can underflow, yet the correct loss is about 2000.
Use log(sum(exp(z-m))) - (z_target-m) directly.
Row a starts at [0,0]. Observe a -> b, with b=1.
Loss is log(2); gradient is [0.5,-0.5].
At learning rate 0.1, what is the updated row?
New row a: [-0.05,0.05].
P(b given a) rises to about 0.524979; loss falls to about 0.644397.
Row b does not change. Target selection is different from input row selection.
Mean cross-entropy divides by position count.
Duplicating all examples preserves the mean gradient.
Clear leaves, use a fresh graph, backward once, update each identity once.
Copy a nonempty seed. Read last-token logits, divide by positive finite temperature, sample, append.
Return seed plus new tokens. Lower temperature sharpens the distribution. Zero is invalid here.
Both abx and cby end with b before different targets.
A bigram model cannot condition on a versus c.
Next: give the classifier an ordered context window.