LLM serving and RAG

A reference librarian asked a hard question does three things in order. She finds the right pages, which can take a minute, she reads them, which takes another minute, and then she writes the answer out by hand, one sentence at a time, while you watch. You judge her on how soon she starts writing and on how fast the sentences come, and you forgive a long answer if the first line arrives quickly.

An assistant built on a language model does the same three things, and they are called retrieval, prefill, and decoding. Memory is where the librarian and the model part ways, because she reads the pages once and remembers them while she writes, at no cost to anyone else. The model must hold its working notes for every conversation it is in the middle of at the same time, and that memory on the machine is the resource that runs out first.

A token is the unit of text the model works in, roughly a short word or part of a longer one, so 750 words is about 1,000 tokens. Prefill is the step where the model reads the whole prompt, which includes the retrieved passages, the instructions, and the question, and it happens once per request. Decoding is the step where the model writes the answer one token at a time, and each new token takes a fresh pass through the model, so a 300 token answer is 300 passes.

During prefill the model stores working notes about every token it has read, and that store is called the key-value cache, or KV cache. Decoding reuses the notes instead of rereading the prompt, which is what makes each step quick, but the notes stay in memory until the answer is finished. A long prompt therefore costs time at the start and a long answer costs time at the end, and both cost memory for as long as the request is active.

Assume decoding produces a token every 20 milliseconds, which is a plausible figure for a mid-sized model on one accelerator, the chip built for model arithmetic, serving several requests at once. A 300 token answer then takes six seconds to finish, and no user waits six seconds looking at a blank box. The design answer is to stream, meaning send each token to the screen as it is produced, so the user sees the first words within about half a second and reads along while the rest arrives. Time to first token and tokens per second are therefore two separate targets, and a service can be fine on one and poor on the other.

Memory decides how many conversations one machine can hold. Assume the model weights take 16 gigabytes and the working notes take half a megabyte per token, both labelled assumptions for this exercise. A request with a 3,000 token prompt and a 1,000 token answer holds 4,000 tokens of notes, or about 2 gigabytes, at its peak. On an accelerator with 80 gigabytes of memory, 64 are left after the weights, which is room for about 32 such requests at once, and the thirty-third waits in a queue. Admission, the decision of whether to accept a new request now, has to count tokens, because a count of requests alone says nothing about memory.

Because requests finish at different times, the server does not wait for a whole batch to end before admitting new work. Continuous batching lets a finished request leave and a waiting one join at every decoding step, so the accelerator stays busy. Paged memory management hands out the working notes in small blocks instead of one large reserved slab per request, which stops a short answer from wasting space reserved for a long one. Both are efficiency tools, and a service still needs hard limits on prompt length, answer length, queued requests, and requests per user.

Prefix caching saves the working notes for a prompt beginning that repeats, such as the same long instruction block at the start of every request, so prefill for the repeated part is skipped. It saves time at the start and nothing during decoding, since the answer is still written fresh. It is also a different thing from a response cache, which stores whole answers and can only be reused when the same question is asked with the same permissions and the same documents. Each cache therefore has its own rule about who may share an entry.

Capacity is measured with the mix of prompt and answer lengths the product will see. A benchmark of short questions with short answers can get through ten times as many requests a second as document questions with long answers on the same machine. The test also includes the queue, the retrieval step, the safety checks, and users who close the window halfway through an answer.

The retrieval half of RAG has its own pipeline that runs before any question arrives. Documents are parsed and cleaned, cut into chunks of a few hundred words each, and tagged with where they came from and who is allowed to see them. Each chunk is turned into an embedding, a list of numbers chosen so passages about the same thing end up close together, and the embeddings are loaded into an index that finds the nearest chunks to a question quickly. When a document is deleted or its permissions change, the index has to change the same day, and that cannot wait for the model to be retrained, because the model was never where the document lived.

When a question arrives, the service checks who is asking and searches the index for the chunks that person is allowed to see. It may reorder them with a small model, then it fits the best few into the prompt under a token limit and asks the language model to answer with references to the chunks it used. Chunk size is the main tuning decision. Chunks of 100 words lose the sentence that explains a fact, and chunks of 2,000 words fill the prompt with text the question did not need. The size is chosen by testing on questions whose supporting passages are known rather than copied from an example.

A wrong answer has two possible causes and they are fixed in different places. If the passage that holds the answer was never retrieved, the problem is the index, the chunking, or the embeddings, and no change to the model will help. If the passage was retrieved and the model still answered wrongly, the problem is in the prompt or the model. The evaluation set therefore needs both measurements, how often the right passage was in the prompt and how often the model answered correctly when it was.

The generation half is graded on whether the answer is correct, whether it is supported by the passages it cites, whether the citations point to the right places, and whether it declines when the evidence is not there. A citation is only a claim of support until a checker has read the passage, and an answer that cites an existing document for a sentence that document does not contain is the failure users notice most. Retrieved text can also carry instructions aimed at the model or content the asker is not allowed to see, so the test set includes both cases.

With the serving paths in place, the remaining question is how to know a week from now that any of them still works, which is where the monitoring section begins.

A language model service gets measured on requests per second. The useful numbers are time to first token, tokens per second during decoding, and how many requests the memory can hold at once, since a long document question and a yes-or-no question are both one request and differ in cost by a factor of fifty.

The two caches get confused with each other. Prefix caching saves the reading of a repeated prompt beginning and still writes a fresh answer, while a response cache returns a whole stored answer.

A response cache then gets keyed on the question alone, when it must be keyed on the asker’s permissions and the document versions as well, and a design that skips that returns one user’s answer to another.

Replacing the vector database, the store that holds the embeddings, is the reflex when the assistant answers wrongly. The fix depends on whether the right passage was in the prompt, and a team that does not log the prompt it sent cannot know.

In a RAG or assistant question this topic takes five to ten minutes and the serving numbers are what separate a strong answer from a vague one. Name the two time targets, first token within half a second and a steady stream after that, and give the memory limit on conversations held at once with one worked number. Then say that retrieval and generation are evaluated separately and offer the chunking and caching details for the deep dive.