Retrieval-Augmented Generation (RAG)

1. Giving the Model Something to Read

A user asks what time their local library closes on Sunday. First, consider an application that sends only the question to the model. It does not look up the library's website or include its opening hours.

The model may have encountered this library's hours during training, but those hours could have changed. It has no current source to check. The response above acknowledges that gap, although a model could also produce a plausible but incorrect time.

With RAG, the application looks up the opening-hours page before asking the model to answer. It sends the relevant passage along with the same question, giving the model the information it needs.

The closing time comes from the passage supplied in this request. The model reads it as part of its input, so there is no need to retrain the model to give it these opening hours.

The three parts of RAG describe this process. Retrieval finds relevant passages, augmentation adds them to the model's input, and generation produces an answer using that input.

A model can read only a limited amount of text in one request, so sending an entire document collection is usually impractical. Retrieval narrows the input to passages that are relevant to the current question.

2. Preparing Documents for Search

Before anyone asks a question, the application needs a collection to search. This collection is often called a knowledge base. It might contain help articles, internal pages, manuals, or files. RAG does not automatically search the internet; it searches the sources the application makes available.

Documents first need to be converted into usable text. A PDF may require text extraction or optical character recognition for scanned pages. Headings, lists, and table labels should survive this conversion because they help explain what the surrounding text means. A table entry such as “4 pm” needs its “Sunday closing time” heading to make sense.

The text is then divided into smaller passages called chunks. These become the units that search returns. A whole manual contains too many topics to be a precise search result, while half a sentence may omit the condition that makes an answer correct.

Real chunking usually considers document structure as well as length. A long section can be split at paragraph or sentence boundaries. Adjacent chunks may repeat a small amount of text, called overlap, so a boundary does not remove all surrounding context. Too much overlap creates duplicate results and uses extra storage.

Length is often measured in tokens, the pieces of text a model processes. A token may be a word, part of a word, or punctuation. The embedding model and the answering model can have different token limits, so a chunk must fit the model that encodes it. There is no single chunk size that suits every manual, table, and code file.

Each chunk should retain its source title, document ID, location, version, and access permissions. These accompanying fields are metadata. They let the application recover the original page, distinguish current information from older versions, and limit retrieval to documents the user is allowed to read.

3. What an Embedding Represents

A user might ask when the library “shuts” while its website lists “opening hours.” Search should be able to connect the question with the relevant passage even when the wording differs.

An embedding model converts text into a vector, which is an ordered list of numbers. It is trained so that useful relationships between texts can be reflected in their vectors. A retrieval model aims to place a question near passages that help answer it, even when they do not share every word.

The document vectors are computed ahead of time. A new question is encoded when it arrives. Both must use compatible encoders that place queries and passages in the same vector space. Some models use different prompts or encoding paths for queries and documents, as described in the Sentence Transformers semantic-search documentation.

An embedding supports comparison, but it is not a copy from which we can reliably reconstruct the original text. The application keeps the passage alongside its vector, or stores a reference that can load it. Search uses the numbers; the answering model will need the actual passage.

4. Finding Relevant Passages

Once the question has a vector, the application compares it with the stored document vectors. It sorts passages by a similarity score and returns the highest-ranked results. Asking for the top k results simply means asking for the first k passages in that ranking.

One common comparison is cosine similarity, which measures how closely two vectors point in the same direction. For vectors whose lengths have been normalized to one, this is their dot product, calculated by multiplying corresponding coordinates and adding the products. The comparison should match the method used to train the embedding model.

For a small collection, comparing the question with every passage is practical. Larger collections usually use an index, a data structure that organizes vectors for faster lookup. Approximate nearest-neighbor search avoids checking every vector, trading some retrieval accuracy for speed. A vector database packages this search with storage, metadata, filtering, and updates; a separate database service is not required for every RAG application.

Where keyword search still helps

Some questions depend on exact text. An error code such as E104, a function name, or a product version can matter more than a broad semantic match. Keyword retrieval, including methods such as BM25, ranks passages using matching terms and their importance in the collection.

Hybrid retrieval combines keyword and vector search. One can preserve exact identifiers while the other helps with paraphrases. Their raw scores have different meanings, so systems often combine ranked lists or use calibrated scoring rather than simply adding the two numbers together.

5. Choosing What Goes into the Prompt

A search for Sunday opening times might return both the hours page and a page about library cards. The hours page contains the closing time needed for this answer. The application selects that passage for the prompt.

A reranker can make a second, more detailed comparison. A common approach uses a cross-encoder, which reads the question and a candidate passage together and assigns a relevance score. Running this model over a small candidate set is more manageable than applying it to every document. The retrieve-and-rerank approach separates fast candidate search from this closer reading.

After ranking, the application removes duplicate passages and selects enough evidence to answer the question. It may include neighboring text when a result refers to a condition explained in the previous paragraph. Document versions and permissions must remain attached throughout this process.

The selected passages become part of the prompt, the input sent to the LLM. In a chat application, that input also includes instructions, the user's question, and often earlier messages. The total must fit within the model's context window, with room reserved for the answer it will generate.

More context is not automatically more helpful. Repeated passages, unrelated results, and conflicting versions can obscure the evidence. Research such as Lost in the Middle also shows that a model's ability to use information can depend on its position in a long input. Context selection should be tested on the questions the application actually receives.

6. One RAG Request, End to End

Document preparation happens before the question arrives and is repeated when the source collection changes. For each request, the application encodes the question, retrieves permitted passages, selects useful evidence, and sends it to the LLM with the question.

Select the public-holiday question to see what happens when the source is incomplete. Search returns the opening-hours page because it concerns visiting the library, but the passage only lists weekdays and Sundays. It gives no public-holiday schedule, so the answer should explain that the information is missing.

7. What Happens Inside the Transformer?

Up to this point, most of the work has happened around the language model. In the common decoder-only setup described here, the assembled prompt now goes through the same Transformer that would process an ordinary chat message. RAG does not require adding a new attention layer or inserting the search index into the model's weights.

First, the answering model's tokenizer converts the instructions, source passages, and question into tokens. Its own embedding layer turns those token IDs into the vectors used inside the Transformer. These token embeddings serve a different purpose from the passage embeddings used for search. The model reads the retrieved text through its normal input path.

During prefill, the model processes the known prompt and computes the representations and attention keys and values for its tokens. During generation, each new token can attend to earlier prompt tokens, including the library hours, as well as earlier answer tokens. The usual attention and feed-forward layers participate in predicting the next token.

No training step is needed for this ordinary inference request. The model's temporary attention state, including its KV cache, should not be confused with a permanent update to its knowledge. A later request needs the relevant text supplied again, unless the application explicitly carries forward or safely reuses the earlier context.

Retrieval-augmented systems can also be trained. The original RAG paper studied models combining a neural retriever with a sequence-to-sequence generator. The retrieve-then-prompt pattern in this article is a common application architecture, rather than a claim that every RAG model has identical internals.

8. Keeping Answers Tied to Their Sources

The cited passage must support the answer's claims. A page listing weekday and Sunday hours does not establish when the library opens on public holidays. Citations let a reader inspect the source, and each claim still needs to agree with the text it cites.

When an answer is wrong, inspect the retrieved passages first. If the relevant rule never reached the prompt, investigate extraction, chunk boundaries, filters, and ranking. If the correct rule was present but the model ignored it, inspect the prompt, conflicting evidence, and generation behavior. Separating these stages makes the problem easier to locate.

Freshness and access belong to the retrieval system

When the library changes its Sunday closing time from 4 pm to 5 pm, the searchable collection needs the updated hours. The new passages need to replace or supersede the old ones, with changed vectors recomputed and stale cached results invalidated where necessary. A RAG application is only as current as the sources and indexes it actually uses.

Access control must be enforced by the application before private text is sent to the model. Asking the LLM not to reveal a restricted document is not a substitute for keeping that document out of an unauthorized request. The same permissions should govern any source link shown to the reader.

Retrieved text is evidence, not an instruction

A document can contain text that tells the assistant to ignore its instructions or expose other information. This is a form of indirect prompt injection. Clearly separating source material from trusted instructions helps, but delimiters and prompt wording alone cannot guarantee safety. Restrict data access and tool permissions outside the model, and test with hostile documents. OWASP's prompt-injection guidance describes why layered controls are needed.

9. How RAG Relates to Fine-Tuning

RAG supplies information for a request, while fine-tuning changes model parameters through training. Retrieval can supply the library's current opening hours, and fine-tuning might teach a consistent response format or improve how the model uses evidence. These techniques can be used together.

Updating a document index is often more direct than retraining whenever a policy changes, and it leaves an identifiable source for the answer. Fine-tuning can improve task behavior, but it does not guarantee exact recall of every fact in its training examples or automatically provide a document citation.

The search mechanism should also match the question. A short uploaded file may fit directly in the prompt. A current account balance needs an authorized lookup against the system of record. A question that spans several manuals may benefit from passage retrieval. RAG is useful when finding the right external material is an important part of answering.

10. Putting the Pieces Together

The application has two connected paths. The preparation path creates searchable passages, while the request path uses them to answer a question. The Python-style sketches below show the responsibilities of each component; the function names are illustrative interfaces, not runnable calls to a particular library.

Prepare the collection

for document in knowledge_base:
    text = extract_text(document)
    chunks = split_into_passages(text)

    for chunk in chunks:
        index.upsert(
            id=chunk.id,
            vector=embed_document(chunk.text),
            text=chunk.text,
            metadata=document.metadata,
        )

Stable chunk IDs allow records to be updated. Source metadata carries the document location and permissions. A complete ingestion process also removes passages from deleted or superseded documents, rather than only appending new ones. Query and document encoders must remain compatible when the embedding model changes.

Answer a question

def answer_question(question, user):
    scope = permissions.allowed_documents(user)
    query_vector = embed_query(question)

    candidates = index.search(
        query_vector,
        allowed_documents=scope,
        limit=20,
    )
    ranked = reranker.rank(question, candidates)
    evidence = pack_context(ranked, token_budget=2000)

    if not evidence:
        return {"answer": "No supporting sources were found.", "sources": []}

    answer = llm.generate(
        instructions=ANSWER_POLICY,
        question=question,
        sources=evidence,
    )
    return {"answer": answer, "sources": evidence}

Here, the permission scope comes from trusted application logic. The search component applies that scope before returning passages. The context-packing function removes duplicates, retains source IDs, and respects the answering model's token budget. Twenty candidates and a 2,000-token evidence budget are examples, not universal defaults.

The answer policy tells the model to support claims with supplied sources and to explain when information is missing. The empty-result check handles one failure case, but a nonempty result can still be insufficient, as the public-holiday example showed. Source support and the model's willingness to acknowledge gaps both need testing.

For a first implementation, use a small collection you can inspect and questions whose answers you know. Read the retrieved passages and the assembled prompt before judging the final answer. That makes it possible to see whether the system found the right information, preserved its meaning, and used it correctly.