Embeddings and Positional Encoding

Course role: v1 is a conceptual bridge with compact scalar exercises. For the cumulative implementation sequence, numerical boundary cases, and learned multi-head projections, continue with deep learning v2. The v1 transformer exercise uses a single attention head even though its API accepts num_heads.

Goal

Transform discrete token IDs into continuous vectors and inject position information so a transformer can reason about order.

Prerequisites: Autograd + NN abstractions.

---

1) Token embeddings

An embedding is a lookup table:

• Vocabulary size: V
• Embedding dimension: D
• Weight matrix: W with shape (V, D)

Lookup for token IDs [t0, t1, ...] returns rows W[t0], W[t1], ....

class Embedding(Module):
    def __init__(self, num_embeddings, embedding_dim):
        k = 1.0 / (embedding_dim ** 0.5)
        self.weight = [[Value(rand(-k, k)) for _ in range(embedding_dim)]
                       for _ in range(num_embeddings)]

    def forward(self, indices):
        return [self.weight[i] for i in indices]

---

2) Sparse gradient updates

The lookup path contributes gradients only to rows selected by the input. Other uses of shared weights, weight decay, or optimizer state can still change unused rows. For this lookup alone:

• embedding.weight[42] gets updated if token 42 appears
• unused rows get no gradient

---

3) Positional embeddings

Unmasked self-attention without position information is permutation-equivariant: permuting input rows permutes output rows the same way. It is not invariant (identical output). Position information lets the model distinguish order; a causal mask additionally restricts which positions are visible.

Learned positional embedding:

• Max sequence length Tmax
• Table P with shape (Tmax, D)
• Positions are indices 0..T-1

pos = position_embedding(list(range(T)))

---

4) Combine token + position

For each position:

input[i] = token_emb[i] + pos_emb[i]

Shapes:

• token_emb: (T, D)
• pos_emb: (T, D)
• input: (T, D)

---

5) Tokenization (this pack)

We use character-level tokenization for clarity:

• Build char_to_idx from the dataset
• Encode text -> list of ints
• Decode ints -> string

The same embedding logic applies to subword tokenization (BPE) used in real GPTs.

---

Key takeaways

1. Embeddings are just a learned lookup table W.
2. Positional embeddings inject order information.
3. The model input is the elementwise sum of token and position vectors.

Next: self-attention, which consumes these (T, D) vectors.

Reconstruct a lookup before the capstone

Set token rows to E0=[1,0], E1=[0,2], and position rows P0=[0.1,0.2], P1=[0.3,0.4]. For IDs [1,1], compute the two combined vectors, then differentiate the sum of all their coordinates with respect to E1.

Check lookup sharing and position addition

The combined vectors are [0.1,2.2] and [0.3,2.4]. They use the same token row but different position rows. E1 receives gradient [2,2], E0 gets [0,0], and each used position row receives [1,1]. Returning fresh detached Value scalars for the lookup would give correct forward numbers but lose those table gradients.

This v1 module has a conceptual checkpoint; its implementation is part of the mini-GPT capstone. For separate executable lookup and position practice before attention, use v2 embeddings. That path also gives tokenization its own exercise instead of treating token IDs as pre-supplied.