How to Build a RAG System from Scratch, One Stage at a Time

How do you build a RAG system from scratch?

You build a RAG system from scratch by writing two paths and testing each stage on its own. The indexing path loads documents, splits them into chunks, embeds each chunk and stores the vectors. The query path retrieves the closest chunks for a question, reranks them, places them in a prompt with their source IDs, asks a language model to answer with citations and checks those citations. TensorTonic has a RAG from Scratch project and a coding problem for most of these stages, and this guide links to them in build order.

RAG stands for retrieval-augmented generation. The language model itself stays unchanged, and the work happens in the code around it, which decides what the model reads before it answers. That is why a RAG system can be built and tested in plain Python, with a model call only at the very end.

What is RAG, and when should you use it instead of fine-tuning?

RAG is a way to answer questions from documents the model was never trained on by retrieving relevant passages and adding them to the prompt, and you should use it when the model needs facts it does not have or facts that change. Fine-tuning changes the weights of the model, so it suits changes to behaviour, such as an output format, a tone or accuracy on a narrow task. The two can be combined, and many production systems fine-tune a model to use retrieved evidence better.

• Choose RAG when the documents are private, large, or updated often, because updating an index is quicker than retraining and every answer can point to its source.
• Choose fine-tuning when prompting with good examples has already failed to produce the behaviour you need.
• Choose neither when the whole source fits comfortably in the prompt, or when the answer comes from a live system such as an account balance, which needs an authorized lookup instead of a search.

The Retrieval-Augmented Generation (RAG) Explained lesson covers this comparison in more depth, and the LoRA lesson explains the most common way to fine-tune when you do need it.

What do you need to know before building a RAG system?

You need comfortable Python, basic NumPy and an understanding of dot products and vector length before you build a RAG system. You do not need to know how to train a neural network, because a RAG pipeline uses an embedding model and a language model that someone else has trained. You do need access to both models, either through a hosted API or as small open models running locally.

• For Python, you should be able to write functions, dictionaries and string processing without looking things up, and this checklist of the Python you need for ML lets you test yourself.
• For NumPy, you should understand array shapes, broadcasting and sorting, which the NumPy study plan covers.
• For linear algebra, you should know what a dot product and a vector norm are, and the linear algebra guide teaches both through code.
• For language models, it helps to know what tokens and a context window are, which the LLM Internals lessons explain.

What are the steps to build a RAG pipeline?

The steps to build a RAG pipeline follow the path of a document into the index and then the path of a question to an answer. Stages one to three prepare the collection ahead of time, stages four to eight run on every request, and stages nine and ten tell you whether the system works and how to fix it. Each stage below has a short explanation and a table that pairs every topic with a page to read and something to practise.

Stage 1: How do you load and clean documents for RAG?

You load documents for RAG by extracting plain text from each file, normalizing it, removing duplicates and attaching metadata such as the source, title, version and access permissions. Text extraction from PDFs and web pages is often the noisiest part of the pipeline, since headers, footers, navigation and broken tables end up mixed with the content. Duplicate and near-duplicate pages waste space in the prompt later, so it pays to remove them here.

Give every document a stable ID at this stage. Later stages pass that ID along with each chunk, and it is what lets an answer cite a source and lets you replace a document when it changes.

Stage 2: How should you chunk documents for RAG?

You should chunk documents into passages of a few hundred tokens with a small overlap, splitting on paragraphs and headings where you can. A retriever compares the question with one chunk at a time, so a chunk that covers several topics dilutes the match, while a chunk that is too small loses the context needed to answer. The overlap repeats a little text across neighbouring chunks so that a fact near a boundary is not cut in half, and the window advances by the chunk size minus the overlap.

Fixed-size windows are the simplest strategy and a sensible first version. Recursive splitting tries paragraph breaks first, then sentences, then words, until each piece fits the limit, and structure-aware splitting keeps sections and tables whole. Measure chunk size in tokens of the embedding model, because that model truncates input beyond its own limit.

Stage 3: How do you create embeddings for RAG?

You create embeddings for RAG by passing every chunk through an embedding model, which returns one vector per chunk, and storing each vector beside the chunk text and its metadata. Text with related meaning produces vectors that point in similar directions, so a question and the passage that answers it end up close together even when they use different words. Questions must be embedded with the same model as the chunks, and some models expect a different prefix for queries and for passages.

Many embedding models produce one vector per token, and the chunk vector is the average of those token vectors with the padding positions left out. Scaling every vector to unit length is worth doing once at indexing time, because the dot product of two unit vectors equals their cosine similarity and is cheaper to compute.

Stage 4: How do you search a vector index for the nearest neighbours?

You search a vector index by embedding the question, scoring it against every stored chunk vector, and returning the k chunks with the highest scores. With unit-length vectors stacked into a matrix, that is one matrix-vector product followed by a partial sort, and this exact search is fast enough for collections of many thousands of chunks. Larger collections use an approximate nearest-neighbour index, which skips most of the vectors and trades a little recall for much lower latency.

The two common approximate indexes work in different ways. An inverted-file index clusters the vectors with k-means and searches only the clusters closest to the question, while HNSW builds a layered graph of neighbours and walks it towards the question. Product quantization and scalar quantization shrink each stored vector so that more of the index fits in memory, at some cost in accuracy.

Stage 5: How do you add keyword search with BM25 and build hybrid search?

You add keyword search by scoring the same chunks with BM25 and then merging its ranked list with the dense ranked list. BM25 rewards chunks that contain the query terms, weights each term by how rare it is across the collection, lets repeated terms count less each time through a parameter usually called k1, and adjusts for chunk length through a parameter usually called b. It finds error codes, product names and rare words that embeddings often blur together.

BM25 scores and cosine similarities sit on different scales, so adding them directly gives one method too much weight. Reciprocal rank fusion avoids the problem by using only positions, giving each chunk the sum of one divided by a constant plus its rank in each list, with the constant commonly set to 60. Hybrid search usually beats either method alone when questions mix exact identifiers with natural language.

Stage 6: How does reranking improve RAG retrieval?

Reranking improves RAG retrieval by running a slower and more accurate model over the top candidates from the first search and reordering them. The first search uses a bi-encoder, which embeds the question and each chunk separately so that chunk vectors can be computed in advance. A cross-encoder reranker reads the question and one chunk together through its attention layers and outputs a single relevance score, which is more accurate and too slow to run on the whole collection.

A common pattern retrieves a few dozen candidates and keeps a handful after reranking. If several of the top chunks repeat the same content, maximal marginal relevance can pick chunks that are both relevant and different from those already chosen.

Stage 7: How do you assemble a RAG prompt with citations?

You assemble a RAG prompt by writing instructions, then the retrieved chunks each labelled with their ID, then the question, while keeping the total inside a token budget. The instructions tell the model to answer only from the supplied passages, to cite the ID of every passage it uses, and to say that it does not know when the passages do not contain the answer. Add chunks in ranked order and stop before the budget is exceeded, leaving room in the context window for the answer.

Adding more context does not always help, because repeated passages, unrelated results and conflicting versions of the same document make the relevant evidence harder to use. Models also tend to use information at the start and end of a long prompt more reliably than information in the middle.

Stage 8: How does the language model generate the answer?

The language model generates the answer by reading the assembled prompt through its normal input path and predicting one token at a time, attending to the retrieved passages as it writes. Nothing about the model changes for RAG, which is why the same question can get a different answer when the documents change. A low temperature keeps the answer close to the evidence, and the application should check the citations in the output before showing it.

The citation check takes only a few lines of code and catches many grounding errors. Parse the bracketed IDs from the answer, confirm that each one belongs to the chunks retrieved for that request, and reject answers that cite nothing or cite an ID that was never supplied.

Stage 9: How do you evaluate a RAG system?

You evaluate a RAG system by measuring retrieval and generation separately on a set of questions whose answers you already know. For retrieval, label the chunks that contain each answer and measure recall at k, which is the share of those chunks found in the first k results, along with precision at k and a rank-aware metric such as NDCG. For generation, check whether the answer is correct, whether every claim is supported by the cited passages, and whether the system declines questions the documents cannot answer.

Overlap metrics such as ROUGE are cheap and only loosely track whether an answer is right, so many teams use a language model as a judge with a written rubric. Check a sample of its verdicts against human labels before trusting it, since model judges favour longer answers and the first answer they see.

Stage 10: What are the common RAG failure modes, and how do you fix them?

The common RAG failure modes are evidence that never reaches the prompt, evidence that reaches the prompt and is ignored, stale documents, and hostile text inside documents. You fix them by debugging in pipeline order, checking the corpus first, then retrieval, then the prompt and the model, since a later stage cannot repair evidence an earlier stage lost. A trace that records the rank, score, source and text of every retrieved chunk makes that order easy to follow.

• When the right passage is missing from the results, look at text extraction, chunk boundaries, metadata filters and the embedding model, and try hybrid search or a reranker.
• When the passage is present and the answer is still wrong, look for conflicting passages, a prompt that is too long, or instructions that do not ask for citations.
• When answers use old information, delete or replace superseded chunks in the index and recompute their vectors, since appending new documents leaves the old ones searchable.
• When a document contains text that tries to give the model instructions, treat retrieved text as evidence only and enforce data access and tool permissions in the application.
• When private text appears in answers for the wrong user, apply permission filters during retrieval, before any text is sent to the model.

Can you build a RAG system without a GPU?

Yes, you can build a RAG system without a GPU, because only the two models do heavy computation and both can run elsewhere. Small open embedding models run on a laptop CPU at a speed that is fine for indexing a few thousand chunks, and exact vector search, BM25, prompt assembly and evaluation are ordinary Python. The language model can be a hosted API or a small local model, which runs on a CPU slowly and is still useful for testing the pipeline.

The coding problems linked in this guide run in the browser and are checked against tests, so there is nothing to install. Our guide on practising ML without a GPU covers where to find compute when you want to run larger models yourself.

How do you put the stages together into a working RAG system?

You put the stages together by writing each one as a small function with a clear input and output, testing it alone, and then calling them in order on a small document you can read in full. The RAG from Scratch project follows this path. You first implement the primitives in plain Python, covering chunking, embeddings, top-k retrieval, context building, citation validation and recall at k, and then you map each one to its LangChain and Chroma counterpart and build the complete pipeline with an embedding model and a chat model.

1. Pick one short document and write five questions whose answers you know, including one the document cannot answer.
2. Chunk the document and print every chunk with its ID to check the boundaries by eye.
3. Embed the chunks, retrieve the top three for each question, and confirm that the answering chunk appears for every answerable question.
4. Build the prompt and read it in full before sending it to the model.
5. Generate answers, validate their citations, and confirm that the unanswerable question gets a refusal.
6. Add BM25, hybrid search and a reranker one at a time, and keep each only if recall at k improves.

What should you build after a RAG system?

After a RAG system, the most useful next step is an agent that decides for itself when to retrieve, followed by an MCP server that exposes retrieval as a tool. An agent calls retrieval as one tool among several in a loop, so it can search again with a better query when the first results are weak. MCP is a protocol that lets an application offer tools and resources to any compatible model client.

The Build a ReAct Agent project covers tool parsing, dispatch, state, control loops, permissions and agent evaluation. The Build an MCP Server project covers JSON-RPC requests, tool schemas, argument checks, dispatch and resources. If you are preparing for interviews, the AI engineer interview questions guide and the LLM interview questions guide both include RAG design questions.

What mistakes do people make when building a RAG system?

• Judging the system only by reading final answers hides where it fails, so print the retrieved chunks for every test question.
• Embedding questions with a different model, or a different version of the same model, from the one used for the chunks makes every similarity score meaningless.
• Forgetting to normalize vectors before taking dot products ranks long vectors first instead of similar ones.
• Dropping the source ID from chunks during indexing makes citations impossible later.
• Adding the raw BM25 score to the raw cosine score lets one method dominate the other, which rank fusion avoids.
• Filling the whole context window with retrieved text raises cost and often lowers answer quality.
• Testing only questions the documents can answer leaves the refusal behaviour untested, so include questions with no answer in the collection.
• Changing the chunk size, the embedding model and the prompt at the same time makes it impossible to tell which change helped.

Practice this

Then build the whole pipeline in the RAG from Scratch project, or continue with the Build a ReAct Agent project.

Related guides

• How to build an LLM from scratch, one component at a time
• LLM interview questions: 43 questions with answers
• NLP interview questions: 35 questions with answers
• AI engineer interview questions
• Which evaluation metric fits which machine learning problem?
• How to build a machine learning portfolio that gets interviews
• How to build an AI agent from scratch, from the ReAct loop to MCP
• How to fine-tune an LLM with LoRA and QLoRA, step by step