Problems

1 / 1

Implement TF-IDF Vectorizer

NLPLinear AlgebraFeature Engineering

Hard

Implement a TF-IDF vectorizer that converts text documents into numerical feature vectors. Return both the TF-IDF matrix and the vocabulary.

Term Frequency (TF):

\text{tf}(t, d) = \frac{\text{count}(t, d)}{\text{total terms in } d}

Inverse Document Frequency (IDF):

\text{idf}(t) = \log\left(\frac{N}{\text{df}(t)}\right)

TF-IDF Score:

\text{tf-idf}(t, d) = \text{tf}(t, d) \times \text{idf}(t)

Where: N = total documents, df(t) = documents containing term t, count(t,d) = occurrences of t in d

Function Arguments

documents: list[str] - List of text documents to vectorize

Loading visualization...

Examples

Input: documents=["the cat sat", "the dog ran"]

Output: matrix shape (2, 5), vocab=["cat", "dog", "ran", "sat", "the"]

Input: documents=["apple banana", "banana cherry"]

Output: matrix shape (2, 3), vocab=["apple", "banana", "cherry"]

Note: "banana" appears in both docs → IDF=0, so it gets zero weight.

Hint 1

Use str.split() and str.lower() for tokenization. Use set() to build vocabulary and sorted() for consistent ordering.

Hint 2

Use Counter for term frequencies and document frequency counting. Use math.log() for IDF calculation.

Hint 3

Use np.zeros() to initialize the matrix. Create a word-to-index mapping with enumerate() for efficient matrix filling.

Requirements

Return tuple: (tfidf_matrix, vocabulary)
tfidf_matrix: np.ndarray of shape (n_docs, n_vocab)
vocabulary: list[str] of unique terms, sorted alphabetically
Tokenize by splitting on whitespace and converting to lowercase
Handle empty documents and empty corpus gracefully

Constraints

Document count ≤ 10,000; vocabulary size ≤ 50,000
Vectorized where reasonable; avoid heavy nested loops
Libraries: NumPy + standard library only

Try Similar Problems

Bag Of Words Bleu Score Word Count Dict Bm25 Cosine Similarity

Prev Next