Embeddings: Look Up Shared Parameters Without Detaching Them

The tokenizer maps symbols to integers. An embedding maps those integers to trainable vectors. The ID itself is not differentiable; the table entries selected by that ID are. This distinction connects discrete text to the scalar autograd engine.

You will implement Embedding, PositionalEmbedding, and combine_embeddings. The key checks are object identity, accumulation for repeated tokens, and matching feature dimensions. The next two language models use token embeddings in different ways; GPT will additionally use positions.

A lookup is a row selection

With vocabulary size V=3 and embedding width D=2, suppose the table is:

E[0] = [ 1,  0]
E[1] = [ 0,  2]
E[2] = [-1,  1]

Input IDs [2,0,2] select three rows, yielding shape (T=3,D=2). The table still owns only V*D=6 parameters. Looking up token 2 twice does not create another trainable vector.

Return references to the original Value objects. Copying the outer Python list is harmless; recreating its scalars with Value(old.data) breaks the path back to the table. This is the lookup equivalent of multiplying a one-hot row by E, without allocating the one-hot representation.

The valid ID range is 0 <= id < V. A negative Python list index selects from the end, but that is not a valid token in this exercise's data contract. Valid tokenization ensures IDs are in range. Initialize each table scalar independently using the specified uniform bound 1/sqrt(D).

Add an address for each position

A learned positional table has shape (Tmax,D). For a sequence of length T, use rows 0 through T-1; require T <= Tmax. Let:

P[0] = [ 0.1,  0.2]
P[1] = [ 0.3, -0.1]
P[2] = [-0.2,  0.4]

Combine the example IDs [2,0,2] with these position rows:

Position	Token row	Position row	Sum
0	[-1,1]	[0.1,0.2]	[-0.9,1.2]
1	[1,0]	[0.3,-0.1]	[1.3,-0.1]
2	[-1,1]	[-0.2,0.4]	[-1.2,1.4]

Addition is feature by feature: both inputs and the output have shape (T,D). Concatenation would instead produce (T,2D) and change every downstream projection's input width. Matching shapes are a precondition; zip will silently truncate unequal lists, so check dimensions while developing.

The same token now has different initial vectors at positions 0 and 2, while sharing its token parameters across both uses.

Trace a sparse gradient update

Define a small diagnostic loss as the sum of every scalar in the combined output. Each used token occurrence contributes derivative 1 to each feature of its row. Therefore:

grad E[0] = [1,1]
grad E[1] = [0,0]
grad E[2] = [2,2]
grad P[0] = grad P[1] = grad P[2] = [1,1]

An unused position row also has zero gradient. This is sparse participation in the graph; our parameter storage and simple SGD iteration are still dense Python lists. In other architectures an output projection tied to the embedding table or an explicit regularizer can give gradients to rows not selected as inputs, so the rule is scoped to this lookup-only loss.

Repeated tokens are another test of autograd accumulation. If row 2 receives only [1,1], inspect overwrite-vs-add behavior. If every table gradient is zero, inspect detached copies.

What position information solves

Without a positional signal or position-dependent mask, self-attention is permutation equivariant: reordering input rows reorders output rows the same way. It is not invariant, which would mean the entire output stayed unchanged. This can be checked directly by permuting Q, K, and V together in the attention equation.

A causal mask already distinguishes which earlier positions are available, so the unrestricted permutation argument does not apply unchanged to masked attention. Learned positions still give this GPT an explicit position-dependent signal. A fixed-window MLP gets order from distinct concatenation slots and does not require this positional table.

The entries begin as random numbers. They acquire useful relationships only through the training objective; lookup alone does not guarantee semantic similarity or analogy arithmetic. Learned absolute positions also do not automatically extend beyond the allocated maximum length.

Reconstruct the contract

Checkpoint 1. For V=5, D=4, and Tmax=8, how many embedding parameters exist regardless of the current sequence length?

Check your reasoning

5*4 + 8*4 = 52.

Checkpoint 2. Why does out[0][0] is table.weight[ids[0]][0] matter for a plain token lookup?

Check your reasoning

both names should refer to the same trainable scalar; gradients from the output must reach that table entry. After adding a positional value, the combined scalar is a new graph node whose parents are those original parameters.

Keep this distinction between parameter identity and activation identity as we turn a lookup table into a complete bigram predictor.