Embeddings: What the Objective Teaches a Vector

An embedding table maps a discrete ID to a trainable vector. Its shape and lookup behavior are guaranteed by the program. Meaningful neighborhoods are an empirical outcome of the training data, objective, optimization, and chosen similarity measure.

From one-hot selection to a row lookup

For vocabulary cat=0, dog=1, bank=2, let the table be:

E = [[ 1, 2],
     [ 2, 1],
     [-1, 0]]          shape (3,2)

The one-hot row [0,1,0] @ E equals [2,1]. Indexing E[1] computes the same selection without constructing a vocabulary-length vector. For token IDs [1,0,1], the output is [[2,1],[1,2],[2,1]], shape (3,2). The table has six trainable scalars; repeated tokens do not create extra parameters.

Dense means the representation is stored as a vector of coordinates, typically much smaller than the vocabulary. It does not mean every coordinate is nonzero or independently interpretable. Embedding dimensions are a design choice, not a fixed 50–300 rule.

Follow a gradient into the table

Suppose the loss is the sum of both coordinates at all three positions above. Each output coordinate receives gradient 1. Row 1 was selected twice, so its gradient is [2,2]; row 0 receives [1,1]; row 2 receives [0,0] from these lookups. At learning rate 0.1, plain SGD makes row 1 [1.8,0.8].

“Unused rows do not update” requires qualification. The lookup contributes zero gradient to unused rows. Weight decay, optimizer state, another use of the table, or tied output weights can still change them. State which path through the model you are analyzing.

What context prediction actually optimizes

Skip-gram uses a center token to predict context tokens; CBOW predicts a center token from context. A simplified skip-gram softmax has an input table E and a separate output table U:

score(center, context) = dot(E[center], U[context])
P(context | center) = exp(score) / sum_over_contexts(exp(score))
loss = -log P(observed_context | center)

For scores [2,0], the first probability is about 0.881 and its loss is about 0.127. Raising the second score to 2 makes that probability 0.5 and loss about 0.693. A favorable score must be assessed relative to competing scores.

The objective pressures input–output dot products, not a rule that every pair of input vectors with related meanings must be close. Words in similar contexts often acquire useful representations, but antonyms can occur in similar contexts too. A low training loss does not establish a semantic guarantee. The original Word2Vec paper describes the predictive architectures; GloVe uses corpus co-occurrence statistics with a different objective.

There is even geometric freedom without changing these scores. For any invertible matrix A, take E' = E A and U' = U A^(-T). Then E' U'^T = E U^T. A nonuniform scaling A can change cosine similarities between rows of E while keeping every score and softmax probability identical. Regularization and training conventions can restrict this freedom, but prediction loss alone does not uniquely identify semantic geometry.

Calculate similarity and its limits

For nonzero vectors a and b:

cosine(a,b) = dot(a,b) / (norm(a) * norm(b))

With cat=[1,2] and dog=[2,1], the dot product is 4, both norms are sqrt(5), and cosine is 0.8. With bank=[-1,0], cosine(cat,bank) is -1/sqrt(5), about -0.447. These are facts about our invented table, not evidence about English.

A zero vector has undefined cosine because the denominator is zero. An API should reject it, skip it, or document a chosen fallback. Positive rescaling leaves cosine unchanged; dot product does not have that property. Choose a metric consistent with the training objective and evaluate it on the target task.

When retrieving neighbors, exclude the query token unless self-matches are desired. Break ties deterministically. A function that includes the query will usually report the token itself first and can disguise poor retrieval quality.

Analogy arithmetic is a hypothesis to test

The query E[king] - E[man] + E[woman] can retrieve queen in some trained spaces, under some candidate sets and similarity measures. It is not an identity or a property enforced by row lookup. Exclude the input words when evaluating the usual analogy task; specify candidates, metric, and held-out questions. Corpus bias and polysemy also affect results. Research on analogy evaluation examines where these tasks succeed and fail.

A static table gives “bank” one row in both “river bank” and “bank account”. A contextual model transforms token vectors using the surrounding sequence, so its output can distinguish those uses. The initial token table remains a lookup table.

Transfer to another task

Product embeddings learned from co-purchases may place accessories near a product even when they are not substitutes. Before interpreting closeness as “similar product,” define whether the downstream task needs complements, substitutes, or shared attributes. Evaluate on that task and account for unseen IDs and changing data.

Pretrained vectors are a possible starting point. Their vocabulary, domain, license, training objective, and evaluation matter more than a promise that all relationships are encoded. A small task can also train its own table.

Reconstruct before moving on

1. E has shape (5,3) and IDs are [2,2,4,2]. What are the output shape and parameter count? (4,3), 15.
2. If each output coordinate receives gradient 1, what is row 2's lookup gradient? [3,3,3].
3. Does a high cosine prove two tokens are synonyms? No; it describes a geometric relation whose usefulness depends on training and evaluation.
4. Can prediction probabilities stay fixed while input-vector cosines change? Yes; the paired invertible table transformations above preserve scores.

Continue to loss functions to make the objective explicit, then to optimization to see how these gradients change parameters.