Problems

1 / 1

Implement TF-IDF Vectorizer

NLPLinear AlgebraFeature Engineering

Hard

Build a TF-IDF representation from text documents. Convert text to lowercase, split on whitespace, and sort the unique vocabulary alphabetically.

\operatorname{tf}(t,d) = \frac{\operatorname{count}(t,d)}{|d|}

\operatorname{idf}(t) = \log\left(\frac{N}{\operatorname{df}(t)}\right)

\operatorname{tfidf}(t,d) = \operatorname{tf}(t,d)\operatorname{idf}(t)

Here, $t$ is a term, $d$ is a document, $|d|$ is its token count, $N$ is the number of documents, and $\operatorname{df}(t)$ is the number of documents containing $t$ . Return a dictionary with tfidf_matrix, a NumPy array of shape $(N,V)$ , and vocabulary, the sorted list of $V$ terms.

Loading visualization...

Examples

Input: documents = ["the cat sat", "the cat ran", "the dog sat"]

Output: {"tfidf_matrix": [[0.135155, 0.0, 0.0, 0.135155, 0.0], [0.135155, 0.0, 0.366204, 0.0, 0.0], [0.0, 0.366204, 0.0, 0.135155, 0.0]], "vocabulary": ["cat", "dog", "ran", "sat", "the"]}

Explanation: The vocabulary fixes the column order, then every document receives one TF-IDF weight per vocabulary term.

Input: documents = ["machine learning is great", "cooking pasta is fun"]

Output: {"tfidf_matrix": [[0.0, 0.0, 0.173287, 0.0, 0.173287, 0.173287, 0.0], [0.173287, 0.173287, 0.0, 0.0, 0.0, 0.0, 0.173287]], "vocabulary": ["cooking", "fun", "great", "is", "learning", "machine", "pasta"]}

Hint 1

Use Counter(tokens) for term counts and Counter.update(set(tokens)) for document frequencies.

Hint 2

Create a token-to-column dictionary with enumerate(vocabulary).

Hint 3

Initialize the output with np.zeros((len(documents), len(vocabulary))).

Requirements

Tokenize with lowercase conversion and whitespace splitting
Sort the unique vocabulary alphabetically
Use unsmoothed natural-log IDF
Return exactly tfidf_matrix and vocabulary in a dictionary
tfidf_matrix must be a NumPy array

Constraints

documents is a nonempty list of nonempty strings
The vocabulary contains at most $50{,}000$ terms
Use NumPy and the Python standard library only

Try Similar Problems

Bag Of Words Bleu Score Word Count Dict Bm25 Cosine Similarity

Prev Next