Text Representation: A Reusable Vocabulary and Weighting Rule

Build a document vector whose coordinates retain their meaning when a new document arrives. You will trace tokenization, a sorted training vocabulary, raw counts, normalized term frequency, document frequency, and the exact IDF rule required by the exercises.

Prerequisites: lists/dictionaries, logarithms, and vector similarity. A logarithm here is simply the inverse of exponentiation; all log calls use the natural logarithm. Knowing log(1)=0 and log(3)≈1.098612 is enough to follow the first trace.

Tokenization is a contract

The exercise uses text.lower().split(). It splits on whitespace and lowercases; it does not strip punctuation, expand contractions, or create an unknown-token symbol. "Red, RED" becomes ["red,","red"], two distinct tokens. More sophisticated tokenization is possible, but changing this rule would change the exercise's coordinates.

Build a sorted vocabulary from training documents. For ["red red blue","blue green",""], the mapping is {"blue":0,"green":1,"red":2}. Sorting makes the index assignment independent of corpus traversal order. At prediction time reuse that mapping; unseen tokens are ignored in counts. No column is appended.

Work the count matrix

Document	Tokens	blue count	green count	red count
red red blue	3	1	0	2
blue green	2	1	1	0
empty string	0	0	0	0

bag_of_words returns one row; transform returns rows in input order; fit_transform returns (matrix,vocab). BoW loses token order: "red blue" and "blue red" have identical vectors. This is an information loss, not a coding error.

A held-out document "red gold" has BoW [0,0,1] with this vocabulary. A document containing only unseen tokens has a zero vector. The zero-vector cosine fallback from the previous exercise is then relevant: there is no direction to compare.

Derive TF, DF, and IDF separately

The TF-IDF exercise uses exactly these definitions:

TF(word,doc) = count(word in doc) / total token count of doc
DF(word) = number of training documents containing word at least once
IDF(word) = log(N / DF(word))
TFIDF(word,doc) = TF(word,doc) * IDF(word)

An empty document has all-zero TF. Its presence still counts in the training document total N. Repeated occurrences affect TF, but a document contributes at most one to a word's DF. Use a set when counting document membership.

For the three training documents above:

Token	DF	IDF	TF in first document	TF-IDF in first document
blue	2	log(3/2)≈0.405465	1/3	0.135155
green	1	log(3)≈1.098612	0	0
red	1	log(3)≈1.098612	2/3	0.732408

A token present in every training document has IDF log(1)=0 under this unsmoothed convention. Replacing this with log(N/(DF+1)) can make an ubiquitous token's IDF negative and does not implement this exercise.

If compute_idf is supplied a vocabulary containing a word absent from a nonempty corpus, the reference contract uses max(DF,1) in the denominator. That is a deliberate finite fallback. Normal fit_transform learns vocabulary from the corpus, so each vocabulary word has positive DF. Empty fit_transform([]) yields ([],{},[]).

Save the vocabulary and IDF together

fit_transform returns (matrix,vocab,idf). Transform a held-out document with compute_tfidf(doc,vocab,idf) using both saved objects; do not refit either on evaluation documents.

For "red gold", there are two tokens even though only red is known. Therefore its TF is [0,0,0.5] and TF-IDF is [0,0,0.549306]. Dividing by the count of recognized words instead would produce a different result. A vector's entries need not sum to one after IDF weighting, and this exercise applies no final L2 normalization.

Terminology alone does not identify a TF-IDF implementation. Raw, normalized, or logarithmic TF; smoothed or unsmoothed IDF; and final vector normalization are independent choices. The Stanford information-retrieval text develops weighting as a design choice; reconstruct the formula your implementation promises instead of substituting a library default.

Complete the missing steps

Training documents are ["sun sun moon","moon"]. Write the sorted vocabulary, DF vector, IDF vector, and TF-IDF row for the first document before revealing the result.

Check the weighting trace

Vocabulary is moon:0,sun:1; DF is [2,1]; IDF is [0,log(2)]. First-document TF is [1/3,2/3], so its weighted row is [0,(2/3)*log(2)], approximately [0,0.462098]. Two occurrences of sun did not turn its DF into 2.

Independent transfer and the next representation

Implement the BoW exercise first. Then implement TF-IDF and verify a repeated word, an empty document, an ubiquitous word, and a held-out unknown word. Predict which returned objects must be saved and what would go wrong if the vocabulary were reordered without reordering IDF.

A high TF-IDF weight means frequent here and uncommon in this corpus under the chosen formula; it does not guarantee semantic importance or factual truth. One-word documents "happy" and "joyful" occupy different coordinates even if their meanings overlap. The embedding module replaces fixed word coordinates with learned vectors and makes the learning signal explicit.