How to Learn NLP from Scratch, One Stage at a Time

How do you learn NLP?

You learn NLP by implementing each method yourself in roughly the order the field developed it, starting with how raw text becomes tokens and ending with the transformer encoders that modern language models grew out of. The stages are tokenization, sparse vectors such as TF-IDF, n-gram language models, word embeddings, text classification, sequence labelling, recurrent models, attention, pretrained encoders such as BERT and the metrics used to judge all of them. TensorTonic has a coding problem with tests for each of these, and the Cracking NLP study plan collects 35 of them into five sections.

This guide works as an NLP roadmap in eleven stages. Each stage has a short explanation and a table that pairs every topic with a page to read and a problem to solve, so you can learn natural language processing from scratch one tested function at a time. The roadmap stops where large language models begin, and the tenth stage hands you over to the guides that cover building and training a GPT-style model.

What does NLP cover today?

NLP today covers every task where a program reads or writes human language, including classification, tagging, search, question answering, translation, summarization and open-ended generation. Most of these are now solved with transformer models, either an encoder such as BERT that turns text into vectors or a decoder that generates text one token at a time. Large language models can handle many of these tasks from a prompt alone, which has made generation the most visible part of the field.

The older methods are still in daily use around those models. Tokenizers decide what every model sees, BM25 and dense embeddings retrieve documents for retrieval-augmented generation, small fine-tuned encoders handle classification and tagging where an LLM call would be too slow or too expensive, and perplexity, BLEU and F1 are still how results are reported. Learning NLP from the beginning gives you the vocabulary to understand why the current models are built the way they are.

What do you need to know before learning NLP?

You need Python, some NumPy, basic probability and a little linear algebra before you start learning NLP. The probability is conditional probability, the chain rule and Bayes theorem, which carry the n-gram and Naive Bayes stages, and the linear algebra is dot products and vector norms, which carry every embedding stage. From the word embedding stage onward you also need to know how a neural network is trained with a softmax, a cross-entropy loss and gradient descent.

• For Python, you should be comfortable with strings, dictionaries and list comprehensions, and the guide to how much Python you need for machine learning lists the rest.
• For neural networks, you should be able to write a small training loop, which the guide to learning PyTorch and the Cracking Deep Learning study plan cover.
• For the maths, the probability study plan and the information theory lessons cover everything the later stages use.

The table lists the maths topics that come up most often in NLP, with a lesson for each and a problem where one exists.

In what order should you learn NLP?

You should learn NLP from representation to modelling, which means tokenization first, then sparse vectors, then probabilistic language models, then learned embeddings, and only then the neural models that use them. Each stage reuses the one before it, since TF-IDF needs a tokenizer, a classifier needs features, a tagger needs a classifier at every position and BERT needs attention. Evaluation metrics come last in this list, although you should start using them as soon as you have a model to score.

Stage 1: How do you turn raw text into tokens?

You turn raw text into tokens by normalizing it, splitting it into units and mapping each unit to an integer ID from a fixed vocabulary. Normalization handles Unicode forms, case, accents and whitespace, and the split can be on words, characters or subwords. Classic pipelines split on words and often reduce each word to a stem or lemma, while modern models use a subword tokenizer such as BPE or WordPiece, which keeps the vocabulary to a fixed size and still covers rare and unseen words.

Byte pair encoding builds its vocabulary by starting from single characters or bytes and repeatedly merging the most frequent adjacent pair, so common words end up as one token and rare words as several. Padding and truncation come at the end of this stage because every batch has to be a rectangle of IDs, and getting the padding mask right here saves debugging later.

Stage 2: How do bag of words and TF-IDF turn documents into vectors?

Bag of words turns a document into a vector with one entry per vocabulary word holding how often that word appears, and TF-IDF scales each count by how rare the word is across the collection. The usual inverse document frequency is the log of the number of documents divided by the number of documents that contain the word, so words such as "the" get a weight near zero and distinctive words get a large one. Both representations ignore word order, and both produce long vectors that are almost entirely zeros.

Cosine similarity compares two of these vectors regardless of document length, which makes TF-IDF a working search engine with only a few lines of code. BM25 improves the ranking by letting repeated words count for less and by correcting for document length, and MinHash estimates the overlap between two documents quickly enough to remove near duplicates from a large corpus.

Stage 3: How does an n-gram language model predict the next word?

An n-gram language model predicts the next word from only the previous n minus one words, using probabilities estimated by counting how often each sequence appears in a corpus. The chain rule of probability writes the probability of a sentence as a product of next-word probabilities, and the Markov assumption cuts each context down to a fixed length so that the counts are manageable. Smoothing then moves a little probability to sequences the corpus never contained, because a single zero would make the probability of a whole sentence zero.

This is the stage where you meet perplexity, which is the exponential of the average negative log probability per token on held-out text and is still the standard intrinsic measure for language models. The same idea of scoring text with a language model drives classic spell correction, which combines an edit distance model of typing errors with a language model of which words are likely.

Stage 4: How do word embeddings like word2vec and GloVe work?

Word embeddings work by learning a short dense vector for each word so that words used in similar contexts end up with similar vectors. Word2vec learns them with a shallow model that predicts the words around a centre word, and skip-gram with negative sampling makes this affordable by scoring each real word pair against a few randomly drawn noise words instead of the whole vocabulary. GloVe reaches similar vectors from the other direction, fitting dot products of word vectors to the logarithm of a global co-occurrence count matrix.

FastText adds vectors for character n-grams, so it can build a vector for a word it never saw during training. All three methods give each word a single vector whatever the sentence, which is the limitation that contextual models such as BERT later remove. Averaging or pooling word vectors gives a simple sentence embedding, and comparing those embeddings with cosine similarity is the core of semantic search.

Stage 5: How do you build a text classifier?

You build a text classifier by turning each document into features and training a model that maps those features to a label such as positive or negative, spam or not spam, or a topic. The standard first model is TF-IDF over words and bigrams with logistic regression, because it trains in seconds, it is easy to inspect and it is hard to beat on short texts with clear keywords. Naive Bayes is the other classic choice, and it applies Bayes theorem to word counts under the assumption that words are independent given the label.

Learn to score a classifier properly at this stage, since most text datasets are imbalanced and accuracy hides poor performance on the rare classes. Precision, recall and F1 for each class, along with a confusion matrix, show you where the model fails and give you a fair number to compare against when you fine-tune an encoder in stage 9.

Stage 6: How do sequence labelling and named entity recognition work?

Sequence labelling assigns a label to every token in a sentence, and named entity recognition is the sequence labelling task that marks spans such as people, places and organizations. NER becomes token classification through BIO tags, where B marks the first token of an entity, I marks a token inside it and O marks a token outside any entity. Part-of-speech tagging uses the same setup with grammatical labels, and it is where hidden Markov models and the Viterbi algorithm come in.

The hard part of this stage is that neighbouring labels depend on each other, so I-PER should never follow O. Choosing each label greedily is the simplest decoder and a good first implementation, and a CRF layer improves on it by scoring transitions between labels and finding the best whole sequence with the Viterbi algorithm. The NLP interview questions guide explains the CRF, the Viterbi recursion and how labels are aligned with subword tokens.

Stage 7: How do RNNs and seq2seq models process text?

RNNs process text by reading one token at a time and updating a hidden state that summarizes everything read so far, which lets them model word order that bag of words throws away. Training unrolls the network over the sequence and runs backpropagation through time, and the gradients shrink or grow as they pass back through many steps. LSTM and GRU cells reduce that problem with gates that control what is kept in the state, and they were the standard NLP models until transformers arrived.

A sequence-to-sequence model uses one recurrent network to encode a source sentence and a second to generate the target one token at a time, which is how neural machine translation first worked. The Cracking Deep Learning plan has a sequence-to-sequence problem for this step. Beam search keeps several partial outputs at each step during generation, and it finds more probable sentences than choosing the single best token every time.

Stage 8: Why did attention and transformers replace RNNs?

Attention and transformers replaced RNNs because they relate every token to every other token directly and process the whole sequence in parallel, which makes long-range dependencies easier to learn and training much faster on GPUs. Attention was first added to seq2seq models so the decoder could look back at every encoder state instead of one fixed summary vector. The transformer, introduced in 2017, kept attention and removed recurrence entirely, computing the softmax of query and key dot products divided by the square root of the key dimension and using it to weight the values.

Multi-head attention runs several attention operations in parallel on smaller projections of the input, and an encoder block wraps it with a feed-forward layer, residual connections and layer normalization. Attention has no sense of order by itself, so positional encodings are added to the token embeddings. For NLP the encoder block is what you need next, and the decoder block with causal masking is covered in the LLM guides from stage 10.

Stage 9: How do pretrained encoders like BERT work?

Pretrained encoders like BERT work by training a stack of transformer encoder blocks on a large amount of unlabelled text and then adapting the same weights to a specific task with a small labelled set. BERT is pretrained with masked language modelling, where some input tokens are hidden and the model predicts them from the context on both sides, and the original model also used next sentence prediction. Because every layer sees the whole sentence, each token gets a contextual vector, so "bank" in a sentence about rivers and "bank" in a sentence about money get different representations.

Fine-tuning adds a small head on top of the encoder. Sentence classification reads the vector at the [CLS] position, token tagging applies a linear layer at every position, and extractive question answering predicts the start and end of the answer span. This is the stage where you can return to your classifier from stage 5 and your tagger from stage 6, fine-tune an encoder on the same data and measure how much it helps.

Stage 10: Where do decoder models and LLMs fit in an NLP roadmap?

Decoder models and LLMs are the point where this NLP roadmap ends and the LLM path begins. A decoder-only model such as GPT uses causal attention, so each token can only attend to the tokens before it, and it is pretrained to predict the next token, which is the n-gram objective from stage 3 with a transformer in place of the counts. Everything you built in stages 1, 3, 7 and 8 carries over directly, from the tokenizer and perplexity to sampling and the attention block.

From here, follow the guide to building an LLM from scratch, which takes a small GPT-style model from tokenizer to training and generation, and use the LLM interview questions guide for attention internals, inference, fine-tuning, alignment and RAG. The LLM internals pages explain each of those topics on its own.

Stage 11: How do you evaluate NLP models?

You evaluate NLP models with a metric that matches the task, which is perplexity for language models, BLEU for translation, ROUGE for summarization and precision, recall and F1 for classification and tagging. BLEU measures how many of the n-grams in the output also appear in a reference, with a penalty for outputs that are too short, while ROUGE measures how much of the reference the output covers. Entity-level F1 for NER counts a predicted entity as correct only when both its span and its type match exactly.

Every one of these metrics has blind spots. Perplexity is only comparable between models that share a tokenizer, and BLEU and ROUGE give no credit to a correct paraphrase that uses different words. Write each metric yourself once, because the implementation shows you exactly what it counts and what it ignores, and read the machine learning evaluation metrics guide for the general trade-offs.

Can you learn NLP without a GPU?

Yes, you can learn NLP without a GPU, because almost every stage in this roadmap runs comfortably on a laptop. Tokenizers, TF-IDF, n-gram models, Naive Bayes, logistic regression and the metrics are all counting and small matrix operations, and word2vec, a small RNN and a small transformer train on a modest corpus with a CPU. A GPU only starts to matter when you fine-tune a full-size pretrained encoder or train a language model on a large dataset.

The coding problems linked in this guide run in the browser and are checked against tests, so there is nothing to install. Our guide on practising ML without a GPU covers where to find compute when you want to train something larger.

What should you build while learning NLP?

You should build one small end-to-end system for every two or three stages, so that the functions you wrote are tested together on real text. A search engine fits after stage 2, a classifier after stage 5, a tagger after stage 6, and a retrieval-augmented question answering system once you have embeddings and a generator. The table pairs each project with the stages it uses and a place to start.

The RAG from Scratch project is a good capstone for this roadmap, since it builds retrieval-augmented generation in plain Python, from chunking and retrieval to a complete assistant, and it puts the retrieval and similarity ideas from stages 2 and 4 to work on a real task. For advice on presenting these projects, read the guide to building an ML portfolio.

What mistakes do people make when learning NLP?

The most common mistakes when learning NLP are skipping the classic stages, using libraries before understanding what they do, and trusting a single number without looking at the outputs. Each one is easy to avoid once you know to look for it.

• Starting with an LLM API skips the tokenization, embedding and evaluation ideas that explain why a model behaves the way it does, so work through at least the first five stages first.
• Calling a library tokenizer or vectorizer without ever writing one leaves you unable to debug unknown tokens, empty documents or a vocabulary that does not match between training and test.
• Fine-tuning a transformer before training a TF-IDF baseline makes it impossible to say whether the larger model was worth its cost.
• Removing stopwords or lowercasing text before a pretrained model throws away negation and casing that the model was trained to use.
• Fitting a vocabulary or IDF weights on the whole dataset leaks information from the test set into training.
• Reporting accuracy on an imbalanced dataset, or token accuracy for NER, makes a weak model look strong.
• Comparing perplexity between models with different tokenizers gives numbers that cannot be compared.
• Reading metric scores without reading model outputs hides errors that a few minutes of looking at examples would show.

Practice this

Then work through the full Cracking NLP study plan, or read the research tracks on Word2Vec, BERT and the Transformer alongside it.

Related guides

• NLP interview questions and answers, grouped by topic
• How to build an LLM from scratch, one component at a time
• LLM interview questions and answers, grouped by topic
• How to learn PyTorch, from tensors to a training loop
• 15 ML papers worth implementing from scratch, and in what order
• Maths for machine learning