Embeddings

One-Hot Problems

• High-dimensional (vocab size)
• Sparse (mostly zeros)
• No similarity captured

Embeddings

• Dense, learned vectors
• Low-dimensional (50-300)
• Similar items = similar vectors

Word2Vec Intuition

• Words in similar contexts → similar vectors
• Skip-gram: predict context from word
• CBOW: predict word from context

Embedding Space

• Similar words cluster
• Relationships are directions
• king - man + woman ≈ queen

Similarity

• Use cosine similarity
• Find nearest neighbors
• Solve analogies with vector arithmetic

Pre-trained Embeddings

• Word2Vec, GloVe, FastText
• Trained on billions of words
• Use them! Don't train from scratch

Beyond Words

• Users, products, images, graphs
• Any discrete entity can be embedded
• Foundation of modern ML