Embeddings: Predict Before Revealing

Row selection

E=[[1,2],[2,1],[-1,0]], IDs [1,0,1].

Predict shape, values, and parameter count.

Answer: (3,2), [[2,1],[1,2],[2,1]], six parameters.

Repeated tokens share parameters

Loss sums every output coordinate.

Row 1 gradient: [2,2]; row 0: [1,1]; row 2: [0,0] from lookup.

Plain SGD at 0.1 changes row 1 to [1.8,0.8].

Weight decay or tied output weights can also affect unused rows.

The objective constrains scores

Two-table model: score(i,j)=dot(E[i],U[j]).

Scores [2,0] give first-class probability ≈0.881. Scores [2,2] give 0.5.

Good prediction does not guarantee synonym neighborhoods in E.

Geometry is not uniquely determined

E'=EA, U'=UA^(-T) for invertible A.

Every input–output score stays the same. Nonuniform scaling can change input-vector cosine.

Which property was actually optimized?

Calculate; then interpret

cosine([1,2],[2,1]) = 4/5 = 0.8.

Zero-vector cosine is undefined. Exclude self-matches when retrieving neighbors.

“High similarity” needs a metric and a task, not just a number.

Transfer check

Co-purchase vectors place printers beside ink. Does this make ink a substitute for a printer?

No. Training signals may encode complementarity. Analogy arithmetic and semantic labels require held-out evaluation.

See the lesson's primary sources and reconstruction exercises.