Embeddings and Positional Encoding

Token embeddings

• Vocabulary size V, embedding dim D
• Weight table W has shape (V, D)
• Lookup by token IDs returns vectors

Sparse updates

The lookup path contributes only to selected rows. Repeated IDs accumulate into the same parameters. Other uses, optimizer state, or weight decay can affect unused rows.

Positional embeddings

• Without positions, unmasked attention permutes outputs with inputs
• This is equivariance, not invariance
• Add learned position vectors P[pos]
• P shape: (Tmax, D)

Combine

input[i] = token_emb[i] + pos_emb[i]

Shape: (T, D)

Tokenization note

We use character-level tokens in this pack for simplicity.